PHP, on a Whim #2: Stop Calling Everything an Array

PHP has one data structure that does basically everything.

$names = ['Mina', 'Ada'];
$scores = ['Mina' => 97, 'Ada' => 91];
$player = ['Mina', 97];
$user = ['name' => 'Mina', 'score' => 97];
$seen = ['Mina' => true, 'Ada' => true];

Those values are doing five different jobs.

1. The first is a list.
2. The second is a dictionary.
3. The third is pretending to be a tuple.
4. The fourth is pretending to be a record.
5. The fifth is pretending to be a set by storing a meaningless true beside every value.

PHP calls every single one of them an array.

This is one of PHP's most convenient features. You learn one data structure, use one literal syntax, and reach for the same standard-library functions almost everywhere.

It is also kind of bullshit.

The runtime has no durable distinction between a list and a dictionary. PHPDoc, Mago, PHPStan, Psalm, and your IDE can describe those values differently:

/** @var list<string> */
$names = ['Mina', 'Ada'];

/** @var array<string, int> */
$scores = ['Mina' => 97, 'Ada' => 91];

/** @var list{string, int} */
$player = ['Mina', 97];

/** @var array{name: string, score: int} */
$user = ['name' => 'Mina', 'score' => 97];

Those annotations are useful. I spend an unreasonable amount of my life making tools understand them.

But PHP itself still sees four values of the same runtime type.

Sometimes that is merely imprecise.

Sometimes it breaks your API.

What the hell is an empty array?#

Imagine an endpoint returning user data:

function return_user_data(array $data): never
{
    echo json_encode([
        'user_data' => $data,
    ]);

    exit();
}

return_user_data([
    'name' => 'Mina',
    'score' => 97,
]);

The result is what you expect:

{
  "user_data": {
    "name": "Mina",
    "score": 97
  }
}

$data has string keys, so json_encode() writes a JSON object.

Now there is no user data:

return_user_data([ /* we have no data about this user */ ]);

The result changes:

{
  "user_data": []
}

Well, fuck.

The API contract did not change. user_data is still meant to be an object. It just happens to contain no fields.

PHP does not have an empty dictionary and an empty list. It only has an empty array.

There are no keys left for json_encode() to inspect, so it has to guess. PHP considers the empty array a list and emits [].

Your frontend expected an object and got an array because the object happened to be empty.

The same PHP type, passed through the same function, under the same response key, changed its JSON type based entirely on its current contents.

You can work around it:

echo json_encode([
    'user_data' => (object) $data,
]);

You can also use JSON_FORCE_OBJECT, although that is a massive hammer that turns actual lists into objects too.

Either way, you are manually restoring information the array itself did not preserve.

The loss also works in the other direction:

$object = json_decode('{}', associative: true);
$list = json_decode('[]', associative: true);

var_dump($object); // []
var_dump($list);   // []
var_dump($object === $list); // true

Once both values become empty PHP arrays, the distinction is gone.

If you encode them again, PHP cannot recover information the runtime value no longer contains.

Even a non-empty list can unexpectedly become an object:

$values = ['first', 'second', 'third'];
unset($values[1]);

echo json_encode($values);

That produces:

{
  "0": "first",
  "2": "third"
}

The value started as a list, but removing one position left a gap in its integer keys. The JSON encoder now decides it looks like an object.

The encoder is not being stupid. It is doing the only thing it can do.

The array never said what it was meant to be.

Hack got into my head#

A large part of the reason this bothers me so much is Hack.

Around 2019-2020, I spent a stupid amount of time playing with Hack and HHVM.

The public community was small. Most of the people I interacted with seemed to be engineers from Facebook, Slack, Quizlet, and a handful of other companies and open-source projects. There was not a massive ecosystem around it, but the little community that existed was fun, and I learned a lot during that period.

I was not only writing tiny Hack experiments either.

I was trying to build Nuxed, a high-performance asynchronous framework written in Hack. It grew into a collection of packages for things like events, filesystems, cryptography, JSON, assets, and the other pieces I apparently thought I should rebuild at the time.

The organization is archived now, but you can still look through it and see what I was up to.

A little over a year ago, I encountered Hack again while working at Quizlet, so this is not only nostalgia for a language I briefly touched years ago and forgot about.

Hack had a huge influence on how I think about type safety and, eventually, language design.

There are also parts of Hack that I think were designed terribly.

Its decision to stop treating every collection as the same kind of array was not one of them.

This will not be the last post in this series that mentions Hack. Its influence is all over Whim, from type-system ideas to syntax choices and basic vocabulary.

Whim could have called its sequential collection a list and its keyed collection a map.

It calls them vec and dict.

You can blame Hack for that.

You can see the same influence in PSL. PHP lists and associative arrays are still PHP arrays there, but the APIs are split into namespaces such as:

use Psl\Dict;
use Psl\Vec;

Hack taught me to think of lists and dictionaries as different things. PSL carried that distinction into PHP at the API level. Whim finally let me carry it into the language and runtime themselves.

What Hack got right#

If you used Hack, especially around version 4.0 and later, these types probably look familiar:

type Names = vec<string>;
type Scores = dict<string, int>;
type Pair = (string, string);
type UniqueNames = keyset<string>;
type User = shape(
  'name' => string,
  'age' => int,
);

The closest PHPDoc equivalents are roughly:

vec<string>                 -> list<string>
dict<string, int>           -> array<string, int>
(string, string)            -> list{string, string}
keyset<string>              -> a set-like list of unique strings
shape('name' => string, ...) -> array{name: string, ...}

The important part was not merely that Hack had shorter names for familiar PHPDoc types.

vec, dict, and keyset are actual Hack value types with different rules.

• A vec is ordered and uses integer positions.
• A dict stores explicit key-value pairs.
• A keyset stores unique integer or string values.

There is some nuance around the other two.

A Hack tuple is a fixed-length but mutable value type. You cannot add or remove positions, but you can replace a value at an existing position.

A Hack shape gives a dict a known field structure, but it is still a dict at runtime, and HHVM does not deeply enforce the field types at runtime.

So it would be wrong to claim that every line above represents a completely separate physical container kind.

That does not weaken the core idea.

Hack stopped asking one runtime type to act as every collection anyone might ever need.

It made the distinction between a sequential collection, a keyed collection, and a set part of the language rather than a PHPDoc convention.

That idea stuck with me.

What Whim stole#

Whim has three concrete array-like runtime values:

$names = vec['Mina', 'Ada'];

$scores = dict[
  'Mina' => 97,
  'Ada' => 91,
];

$player = ('Mina', 97);

Those are a vec, a dict, and a tuple.

They are not three spellings for the same underlying Whim value.

The VM represents them with separate value kinds. The actual Rust enum is larger, but the relevant part looks like this:

pub(crate) enum ValueKind {
    // ...
    Vec,
    Dict,
    Tuple,
    // ...
}

The Value representation also stores separate payloads for VecObject, DictObject, and TupleObject.

This is not parser sugar.

At runtime, a vec is a vec, a dict is a dict, and a tuple is a tuple.

They have different operations, different mutation rules, and different type relationships.

Three values, three promises#

Splitting collections only matters if the values actually promise different things.

Whim's do.

Vecs#

A vec is an ordered mutable collection with dense integer positions starting at zero:

$names = vec['Mina', 'Ada'];

$names[1] = 'Grace';
$names[] = 'Linus';

assert!($names == vec['Mina', 'Grace', 'Linus']);

You cannot create arbitrary holes in it. Indexed assignment replaces an existing position, and appending adds the next position.

That means a vec cannot quietly stop being a list because somebody removed or inserted the wrong key.

Whim arrays are value types. Vecs use copy-on-write (🐄) storage internally, so assigning one vec to another variable does not make ordinary mutation change both values:

$first = vec[1, 2];
$second = $first;

$second[] = 3;

assert!($first == vec[1, 2]);
assert!($second == vec[1, 2, 3]);

The runtime may share storage until one side changes, but the language-level behavior is value semantics.

Dicts#

A dict is an ordered mutable keyed collection:

$scores = dict[
  'Mina' => 97,
  'Ada' => 91,
];

$scores['Grace'] = 94;

Whim dict keys may be booleans, integers, or strings.

I'm debating allowing other types as keys, but as of now, only bool, string, and int may be used as dict keys, this could be false in future Whim versions.

More importantly, keys do not silently change type.

These are three different keys:

$values = dict[
  1 => 'integer',
  '1' => 'string',
  true => 'boolean',
];

assert!(length!($values) == 3);

Try the equivalent in PHP:

$values = [
    1 => 'integer',
    '1' => 'string',
    true => 'boolean',
];

var_dump($values);

PHP converts '1' to the integer key 1, then converts true to the integer key 1, so every entry overwrites the previous one.

You end up with:

[
    1 => 'boolean',
]

PHP's array key conversion rules are historical, familiar, and occasionally useful.

They are also awful if you are trying to build a strict runtime type system.

In Whim, a key keeps its type.

Allowing boolean keys also enables types like this:

type Partition<T> = dict[
  true => vec<T>,
  false => vec<T>,
];

$numbers = dict[
  true => vec[2, 4, 6],
  false => vec[1, 3, 5],
];

assert!($numbers is Partition<int>);

Could that be represented with string keys such as 'yes' and 'no'? Of course.

But if the domain is actually boolean, I would rather let the key remain boolean than stringify it to make the container happy.

Tuples#

A tuple represents a fixed structure:

$player = ('Mina', 97);

assert!($player[0] == 'Mina');
assert!($player[1] == 97);
assert!($player is (string, int));

Whim tuples are immutable.

This is rejected:

$player[1] = 100;

A tuple currently contains between one and twelve fixed items. Each position may have a different type, and a one-item tuple needs a trailing comma:

$only = ('value',);

Hack's tuples are mutable at their existing positions. Whim's are not.

In Whim, the distinction is part of what the tuple means. A tuple is a fixed piece of structure. A vec is a collection you can edit, append to, and remove from.

Even when they contain the same values, they are not the same kind of thing:

$tuple = ('Mina', 97);
$vec = vec['Mina', 97];

The first is an immutable tuple.

The second is a mutable vec that currently happens to have two entries.

Hack also has keyset. Whim currently does not.

There is no deep philosophical speech hiding here. Whim has no built-in keyset value today, and I am not going to invent a profound reason for that omission. Maybe i will add it someday, maybe not 🤷‍♂️

The operations should tell the truth too#

Making vecs and dicts separate values also means their operations can describe what they actually do.

Consider removing an item from a vec:

$values = vec['a', 'b', 'c', 'd'];

$removed = remove!($values, 1);

assert!($removed == 'b');
assert!($values == vec['a', 'c', 'd']);

remove!() preserves order. Every item after the removed position shifts one place to the left, so the vec remains dense.

That is the behavior you normally want from something claiming to be a list, but shifting values takes time proportional to everything after the removed position.

Sometimes you do not care about order:

$values = vec['a', 'b', 'c', 'd'];

$removed = swap_remove!($values, 1);

assert!($removed == 'b');
assert!($values == vec['a', 'd', 'c']);

swap_remove!() fills the hole with the last item instead. It can be constant time when the vec's storage is not shared, but it changes the order.

The language makes you choose.

Do you want stable order, or cheap removal?

PHP's unset($values[1]) makes a different choice. It removes the integer key and leaves the surrounding keys alone. A value that was being used as a list can now have a hole, and, as we saw earlier, that can even change how it is encoded to JSON.

Membership is explicit too:

$values = dict[
  'name' => 'Mina',
  'nickname' => null,
];

assert!(contains!($values, 'Mina'));
assert!(contains_key!($values, 'name'));
assert!(contains_key!($values, 'nickname'));
assert!(!contains_key!($values, 'missing'));

contains!() searches values.

contains_key!() checks whether a key exists, even if the corresponding value is null.

These are small operations, but they follow the same rule as the collection types themselves: do not make the runtime guess what the programmer meant.

Shapes are types, not more containers#

Whim has three concrete array value kinds, but its structural type system can describe many more structures.

A homogeneous vec type describes the type of every value:

type Names = vec<string>;

A homogeneous dict type describes its keys and values:

type Scores = dict<string, int>;

These checks are enforced at runtime:

function total(vec<int> $values): int {
  $result = 0;

  foreach ($values as $value) {
    $result += $value;
  }

  return $result;
}

total(vec[10, 20, 30]);

Whim can also describe individual positions in a vec:

type ResultRow = vec[
  string,
  int,
  ...float
];

That means:

1. position zero must be a string;
2. position one must be an integer;
3. any remaining positions must be floats.

A dict shape describes required literal keys:

type User = dict[
  'name' => string,
  'age' => int,
];

Without a rest entry, that shape is exact. The dict must contain exactly those keys.

A bare rest entry allows additional keys:

type Named = dict[
  'name' => string,
  ...
];

A typed rest entry can constrain them:

type ScoredUser = dict[
  'name' => string,
  ...<string, int|float>
];

Now the dict must have a string name, while any additional entries must use string keys and integer or float values.

These are not new runtime container kinds.

A value satisfying dict['name' => string, 'age' => int] is still a dict. A value satisfying vec[string, int] is still a vec.

The shape tells the runtime what structure that vec or dict must have.

That distinction also explains why Whim has both tuple types and vec shapes:

type TuplePair = (string, int);
type VecPair = vec[string, int];

Both describe two ordered positions.

TuplePair accepts an immutable tuple.

VecPair accepts a mutable vec of exactly two values.

The element types may look identical, but the value still carries a different promise.

An empty vec has the element type never, so it fits vec<T> for any T. An empty dict similarly has key and value types of never, so it fits any valid homogeneous dict type.

That does not erase the distinction between them.

An empty vec is still a vec.

An empty dict is still a dict.

Empty no longer means "guess"#

Whim's JSON data model makes the distinction explicit:

namespace Whim\Json;

type Value =
  null
  | bool
  | int
  | float
  | string
  | vec<Value>
  | dict<string, Value>;

1. A JSON array is represented by a vec.
2. A JSON object is represented by a dict.

These two values remain different even when both are empty:

dict['user_data' => vec[]]

represents:

{
  "user_data": []
}

while:

dict['user_data' => dict[]]

represents:

{
  "user_data": {}
}

The encoder does not need to inspect the current keys and guess what you meant. You already told it when you created the value.

The decoder can preserve the distinction for the same reason. An empty JSON array becomes an empty vec. An empty JSON object becomes an empty dict.

This is where separate runtime collection kinds stop looking like type-system decoration.

They preserve information PHP's array cannot represent.

But sometimes I really do mean any array#

Splitting one type into several creates an obvious problem.

What if a function genuinely does not care whether it receives a tuple, vec, or dict?

Whim has a common type for that:

array<K, V>

For example:

function count_values(array<_, _> $values): int {
  return length!($values);
}

assert!(count_values(vec[1, 2]) == 2);
assert!(count_values(dict['one' => 1]) == 1);
assert!(count_values((1, 'two')) == 2);

array<K, V> accepts a tuple, vec, or dict whose keys fit K and whose values fit V.

It is a read-only type view.

It does not convert the value into some fourth generic array representation. The runtime value is still whichever concrete kind was passed in.

The read-only part matters because mutation means different things for each kind.

Appending is valid for a vec.

Adding an arbitrary key is valid for a dict.

Neither operation is valid for a tuple.

Code working through the common array<K, V> view may read, iterate, and inspect the collection without pretending that every mutation makes sense for every concrete kind.

So, yes, Whim still uses the word array.

The title is not asking us to ban the word.

Stop calling every concrete collection an array.

Call something an array when you genuinely mean that any array-like collection is acceptable.

Runtime types mean runtime work#

In PHPDoc, checking list<int> is somebody else's problem.

A static analyzer proves what it can before the program runs, then PHP receives a normal array.

Whim enforces collection types at runtime.

That means checking a vec<int> may require inspecting every value. Checking a dict shape may require validating its required keys, rejecting extra keys, and checking each corresponding value. Nested structural types can require more work again.

Doing all of that from scratch every time a value crosses a typed boundary would be stupidly expensive.

So the runtime caches structural array checks.

The current implementation has states conceptually like this:

enum ArrayTypeCheck {
    Unknown,
    Clean(ArrayTypeCheckId),
    Dirty {
        id: ArrayTypeCheckId,
        slot: u32,
    },
}

After a vec or dict successfully satisfies a structural type, the runtime can remember that result.

If one slot changes, the cache records which slot became dirty. A later check can often focus on the changed value instead of walking the whole collection again.

If mutations become too complicated to track safely, the cache is invalidated and the runtime falls back to a full check.

Tuples have their own simpler cache because the tuple's own fixed positions cannot be replaced.

You can see the current implementation in the runtime's array.rs, vec.rs, dict, and tuple.rs code.

This is one of those places where a syntax example hides most of the work.

Writing this:

function read_user(dict['name' => string, 'score' => 0..=100] $user): void {
  // ...
}

is easy.

Making that check correct, reasonably fast, compatible with mutation, and safe around nested values is the actual feature.

Splitting arrays does not make everything easier#

There is a cost to making these distinctions real.

Every API has to tell the truth about what it wants.

• Does a function require a vec because order and dense positions matter?
• Does it require a dict because keys matter?
• Does it return a tuple because the number and meaning of its positions are fixed?
• Can it accept any array through array<K, V>?

That is more work than writing array everywhere and letting callers figure it out.

The standard library also needs APIs that preserve those distinctions. Whim has separate Whim\Vec and Whim\Dict namespaces for the same reason PSL does. Operations need to be clear about whether they preserve keys, discard keys, return a sequential result, or build a keyed result.

Conversions become explicit too.

That can feel annoying when you have a dict and a function asks for a vec, but the annoyance is useful. It forces the API boundary to say whether keys are meaningful.

The runtime has more value kinds to implement and optimize.

The type system needs subtyping and runtime-checking rules for homogeneous collections, shapes, tuples, rest entries, and the common array view.

JSON, equality, iteration, spreading, destructuring, reflection, serialization, and pattern matching all need semantics for each kind.

This design is not free.

I still think the result is worth it in Whim.

The value knows what it is, and code does not need to reverse-engineer intent from whichever keys happen to exist at that moment.

Could PHP do this?#

Technically, yes.

Sensibly and compatibly, that is a much harder question.

Replacing PHP's existing array with separate vec and dict values is not realistic.

PHP code depends on current array behavior everywhere. Code mixes integer and string keys, relies on automatic key conversion, creates gaps, appends after explicit keys, passes arrays through extensions, serializes them, mutates them through references, and feeds them to thousands of functions that expect the existing runtime representation.

Changing what [] means would break the world.

The more plausible approach would be additive.

PHP could keep array exactly as it works today and introduce new value kinds beside it:

$names = vec['Mina', 'Ada'];

$scores = dict[
    'Mina' => 97,
    'Ada' => 91,
];

$player = ('Mina', 97);

That avoids changing existing array literals, but it immediately creates a pile of questions.

1. Does a vec satisfy an array parameter?
2. If yes, every internal function and extension accepting an array must understand the new representation or receive an automatic conversion.
3. If no, most existing PHP APIs reject the new values, which makes adoption painful.
4. What does is_array() return?
5. What do array_map(), array_filter(), array_merge(), and hundreds of other array functions accept and return?
6. Does json_decode(..., associative: true) return vecs and dicts, or does that need a new mode?
7. Does json_encode() preserve the new kinds while retaining the old guessing rules for legacy arrays?
8. How do serialize(), var_export(), reflection, unpacking, variadics, references, and extension APIs represent them?
9. Are tuples immutable?
10. Are shapes runtime-enforced, statically analyzed, or both?
11. Can old arrays be converted automatically?

If PHP tries to infer whether an old array should become a vec or dict from its keys, we are back at the original problem. An empty array provides no answer.

Then there is the engine itself.

PHP's existing array is deeply integrated into Zend Engine and extension APIs. New runtime value kinds would need new VM operations, garbage-collection behavior, copying rules, comparison semantics, debug output, C APIs, optimizer support, and long-term maintenance.

This is not one RFC. It is a project.

Would I like PHP to have real vecs, dicts, tuples, and runtime shapes?

Yeah.

I think APIs would become clearer, JSON boundaries would become safer, and a lot of information currently trapped in PHPDoc could become part of the language.

Do I have a magical migration plan that makes it easy?

Absolutely not.

This is another example of a design that works nicely in Whim partly because Whim has nothing to remain compatible with.

The idea may be good.

Adding it to PHP would still be brutal.

Hack was right about this one#

Hack is not a perfect language.

There are decisions in Hack I dislike, decisions Whim deliberately avoids, and decisions that will probably get their own posts later in this series.

But Hack permanently changed how I think about collections.

A list and a dictionary are not the same thing merely because one runtime can squeeze both into the same data structure.

A tuple is not just a list whose author hopes nobody appends to it.

A shape is not merely a comment explaining which keys should probably exist.

An empty list and an empty dictionary are different values.

The runtime should not wait until JSON encoding to guess which one you meant.

Whim borrows that basic lesson, then makes its own choices around immutable tuples, boolean dict keys, strict key identity, runtime-enforced shapes, and a common read-only array view.

Maybe PHP can adopt pieces of that someday.

Maybe the compatibility cost is too high.

Either way, we should at least be honest about the problem.

PHP's array is convenient as hell.

It is also doing far too many jobs.

Stop calling everything an array.