24
Lukas Atkinso: Net-Negative Cursor
(lukasatkinson.de)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 01 Jun 2025
24 points (90.0% liked)
Programming
20929 readers
30 users here now
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
- Follow the programming.dev instance rules
- Keep content related to programming in some way
- If you're posting long videos try to add in some form of tldr for those who don't want to watch videos
Wormhole
Follow the wormhole through a path of communities [email protected]
founded 2 years ago
MODERATORS
In UTF8, all bytes that are not an ASCII character have the high bit set.
This person Unicodes
You are right with this. But still, in Rust, a vector of u8 is different from a sequence of unicode characters. This would not work in Python3 either, while it'd work in Python2.
~~yeah it's incorrect bc it destroys multibyte characters, but~~ no idea what you're saying about u8 being a different type from unicode. the original code was reading bytes and converting them too? the typing isn't the issue, you can still store utf8 as a series of bytes
It doesn't. As the poster two levels up said, all bytes that don't represent an ASCII character have the high bit set, even the follow-up bytes in multibyte sequences. So the condition
b >= 32will match and preserve them.yeah fair enough. that wasn't really my point and I wasn't paying attention
No, the filter is correct even for UTF-8. Any ASCII character is exactly unchanged in UTF-8 (part of the reason it is popular). Since this code only filters out ASCII characters it works fine with ASCII or UTF-8.