But if you need your data to be in that new encoding, losing data can be better than things being broken. You have no idea which bytes were replaced by ?. Unfortunately, when you replace characters with encode, you might lose information. By default, that replacement character is ?. The invalid and undef options replace characters that can’t be translated with a different character. encode ( "Windows-1252", invalid: :replace, undef: :replace ) => "hi?" You can work around this error if you pass extra options into encode: irb(main):064:0> "hi∑".
You’ll see that error when a character in one encoding doesn’t exist in another, or when Ruby can’t figure out how to translate a character between two encodings. Most encodings are small, and can’t handle every possible character.
encode ( "Windows-1252" ) Encoding::UndefinedConversionError: U+2211 to WINDOWS-1252 in conversion from UTF-8 to WINDOWS-1252 Changing the encoding changed how the string printed, without changing the bytes.Īnd not all strings can be represented in all encodings: irb(main):006:0> "hi∑". bytes => # What would that string look like interpreted as ISO-8859-5 instead? Take a look at what a single set of bytes looks like when you try different encodings: # Try an ISO-8859-1 string with a special character! And a string’s encoding defines that relationship. But there’s still a relationship between bytes and characters. Instead of one byte, ṏ is represented by the group of bytes. Now it’s harder to tell which number represents which character. It gets trickier when you use characters that are less common in English: irb(main):002:0> "hellṏ!". In this encoding, 104 means h, 33 means !, and so on. You can think of a string as an array of bytes, or small numbers: irb(main):001:0> "hello!". If you can imagine what encoding does to a string, these bugs are easier to fix. So, when you have a bad encoding, how do you figure out what broke? And how can you fix it? What is an encoding? Or maybe “they’re” starts showing up as “they’re”. When you check your exception tracker and see Encoding::InvalidByteSequenceError: "\xFE" on UTF-8 You only really think about a string’s encoding when it breaks.
0 kommentar(er)
