C++17: codecvt_utf8 is deprecated

Last week, I got this compilation error after upgrading my Visual C++ project to C++17 for using codecvt_utf8 to convert UTF-8 string back and fro wchar_t string.

error C4996: ‘std::codecvt_utf8<wchar_t,1114111,0>’: warning STL4017: std::wbuffer_convert, std::wstring_convert, and the <codecvt> header (containing std::codecvt_mode, std::codecvt_utf8, std::codecvt_utf16, and std::codecvt_utf8_utf16) are deprecated in C++17. (The std::codecvt class template is NOT deprecated.) The C++ Standard doesn’t provide equivalent non-deprecated functionality; consider using MultiByteToWideChar() and WideCharToMultiByte() from <Windows.h> instead. You can define _SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING or _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS to acknowledge that you have received this warning.

The workaround as stated in error is to define _SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING or _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS. The offending code in question is below.

#include <locale>
#include <codecvt>

std::string utf8_encode(const std::wstring& source)
{
	return std::wstring_convert<std::codecvt_utf8<wchar_t>>().to_bytes(source);
}

std::wstring utf8_decode(const std::string& source)
{
    return std::wstring_convert<std::codecvt_utf8<wchar_t>>().from_bytes(source);
}

Or if you can live with platform specific code or coding for Windows platform exclusively, you can use MultiByteToWideChar() and WideCharToMultiByte() as suggested in this StackOverflow post.

// Convert a wide Unicode string to an UTF8 string
std::string utf8_encode(const std::wstring &wstr)
{
    if (wstr.empty()) return std::string();
    int size_needed = WideCharToMultiByte(CP_UTF8, 0, &wstr[0], (int)wstr.size(), NULL, 0, NULL, NULL);
    std::string strTo(size_needed, 0);
    WideCharToMultiByte(CP_UTF8, 0, &wstr[0], (int)wstr.size(), &strTo[0], size_needed, NULL, NULL);
    return strTo;
}

// Convert an UTF8 string to a wide Unicode String
std::wstring utf8_decode(const std::string &str)
{
    if (str.empty()) return std::wstring();
    int size_needed = MultiByteToWideChar(CP_UTF8, 0, &str[0], (int)str.size(), NULL, 0);
    std::wstring wstrTo(size_needed, 0);
    MultiByteToWideChar(CP_UTF8, 0, &str[0], (int)str.size(), &wstrTo[0], size_needed);
    return wstrTo;
}

For portable code, you can use the UTF8-CPP library by Nemanja Trifunovic. Note: we have to check for _MSC_VER macro to detect Windows platform because on Windows, wchar_t is UTF-16 while on other platform such as Linux and MacOS, wchar_t is UTF-32! To get the code example compiled successfully, an old version of UTF8-CPP have to be used instead of the latest version listed above.

// Convert a wide Unicode string to an UTF8 string
std::string utf8_encode(const std::wstring &wstr)
{
    std::string utf8line;

    if (wstr.empty())
        return utf8line;

#ifdef _MSC_VER
    utf8::utf16to8(wstr.begin(), wstr.end(), std::back_inserter(utf8line));
#else
    utf8::utf32to8(wstr.begin(), wstr.end(), std::back_inserter(utf8line));
#endif
    return utf8line;
}

// Convert an UTF8 string to a wide Unicode String
std::wstring utf8_decode(const std::string &str)
{
    std::wstring wide_line;

    if (str.empty())
        return wide_line;

#ifdef _MSC_VER
    utf8::utf8to16(str.begin(), str.end(), std::back_inserter(wide_line));
#else
    utf8::utf8to32(str.begin(), str.end(), std::back_inserter(wide_line));
#endif
    return wide_line;
}
Advertisement

2 responses to “C++17: codecvt_utf8 is deprecated”

  1. Hi thanks for sharing this. Is there a way of doing this with the C++17 standard library without using deprecated API?

    Like

    1. Not that I am aware of.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: