BlueCloth turns UTF8 into ASCII-8BIT

[Phrogz]

C:\>type test_encoding.rb
Encoding.default_external = 'UTF-8'

require 'bluecloth'
utf8 = IO.read('hi.utf8')
out  = BlueCloth.new(utf8).to_html
p utf8.bytes.to_a, utf8.encoding, out.encoding,
  BlueCloth::VERSION, RUBY_VERSION, RUBY_PLATFORM

C:\>ruby -KU test_encoding.rb
[226, 128, 156, 104, 105, 226, 128, 157]
#<Encoding:UTF-8>
#<Encoding:ASCII-8BIT>
"2.0.5"
"1.9.1"
"i386-mingw32"

This is the pure Ruby version of BlueCloth installed on Windows 7/64-bit.

[Phrogz]

Now with formatting. Sorry.

C:\>type test_encoding.rb
Encoding.default_external = 'UTF-8'

require 'bluecloth'
utf8 = IO.read('hi.utf8')
out  = BlueCloth.new(utf8).to_html
p utf8.bytes.to_a, utf8.encoding, out.encoding,
  BlueCloth::VERSION, RUBY_VERSION, RUBY_PLATFORM

C:\>ruby -KU test_encoding.rb
[226, 128, 156, 104, 105, 226, 128, 157]
#<Encoding:UTF-8>
#<Encoding:ASCII-8BIT>
"2.0.5"
"1.9.1"
"i386-mingw32"

[deveiant]

I'll add a test case and a fix here shortly. Thanks for the report!

[Phrogz]

Replying to deveiant:

I'll add a test case and a fix here shortly. Thanks for the report!

Thanks, that would be great. This is currently forcing me to recommend Textile over Markdown for an internal documentation project starting in a few days. (And I fear the adoption may suffer greatly due to the HTML-centric nature of Textile markup.)

[deveiant]

Fixed in r3Ae04eccd695ee.

[deveiant]

My fix was causing a segfault, so I'll do a bit more research on how to properly do m17n from the Ruby C API and re-release a fix.

[Phrogz]

Ping on this...?

[deveiant]

I have a good handle on what I need to do from Ruby, but I'm still testing what Discount will actually support. I also need to update to Discount 1.6.2, which was released last week so I'll roll this into that work.

[Phrogz]

Ping: as I'm coming up on a new release, I'd like to know when this might be fixed to know if I should be re-including BlueCloth support.

[styx]

As rails 3.0 are out and ruby 1.9.2 should be used this bug became an issue for me (as I in update progress). Looking forward for fix. Thanks.

[deveiant]

Fixed in r055519ec5f783a48262a58cf6300f27737992f02.