The tokenizer is capable of decoding spaceless tokens into compound words following a set of rules referred to as a grammar in Natural Language Processing (NLP). I do LLM research and have spent an uncomfortable amount of time staring at the encoded outputs of most tokenizers when debugging. Normally spaces are not included.

There is of course a token for spaces in special circumstances, but I don’t know exactly how each tokenizer implements those spaces. So it does make sense that some models would be capable of the behavior you find in your tests, but that appears to be an emergent behavior, which is very interesting to see it work successfully.

I intended for my original comment to convey the idea that it’s not surprising that LLMs might fail at following the instructions to include spaces since it normally doesn’t see spaces except in special circumstances. Similar to how it’s unsurprising that LLMs are bad at numerical operations because of how the use Markov Chain probability to each next token, one at a time.

This is because spaces typically are encoded by model tokenizers.

In many cases it would be redundant to show spaces, so tokenizers collapse them down to no spaces at all. Instead the model reads tokens as if the spaces never existed.

For example it might output: thequickbrownfoxjumpsoverthelazydog

Except it would actually be a list of numbers like: [1, 256, 6273, 7836, 1922, 2244, 3245, 256, 6734, 1176, 2]

Then the tokenizer decodes this and adds the spaces because they are assumed to be there. The tokenizer has no knowledge of your request, and the model output typically does not include spaces, hence your output sentence will not have double spaces.

Thanks, it’s definitely a good escape from the code, that’s for damn sure.

Old cars are work for sure, but if you are willing to learn it’s not bad.

I have a 2007 Mustang. I’ve replaced the entire front suspension, rear differential, alternator, and paid an upholsterer to replace the convertible top. I upgraded the radio and put in a 10inch touch screen with Wireless carplay and integrated backup camera. Next up is dropping the trans to replace the clutch plate, throw-out bearing, resurface the flywheel, and replace the rear main seal on the engine while I’m down there because the flywheel is rusty and accumulates a thin layer of rust every morning that makes a grinding noise for 30 seconds until it grinds off.

It definitely doesn’t just work like a new car, but since I do the work myself it also doesn’t cost me much.

Absolutely air traffic in the sky should be identified. There is no problem with that, but it’s the idea that it is too easy to find out everything about an aircraft owner by simply seeing the number on their tail.

The rich guys obfuscate that info with shell corps to own the aircraft.

Shouldn’t everyone have the right to the same level of privacy regardless of how much money they have?

No you cannot. You cannot easily find someone’s address from looking at their plate. You need more information, or to do some advanced searching. It is simply not the same.

It is different because you typically need to know the municipality I live in first.

Also the registration allows anyone to track me anytime I fly.

How would you feel if you had a public gps transponder on your car publicly showing who you, where you are, and where you live? Also what if you are required to plaster that registration number on the side of your vehicle in large letters that can be seen from a block away?

It’s a massive invasion of personal privacy.

Shitposters ride for free

This is actually most helpful to the little guys that own $20,000 airplanes.

I have a small airplane and it’s always bothered me that my name and address are publicly accessible through the FAA registry.

Most pilots I know are careful about photos they publish online showing their tail number printed in large bold letters on either side of the aircraft. This registration number can be entered into websites like flightaware.com and someone is literally two clicks from seeing my full name and home address.