Welcome to Joel Lipman .Com

Preparing Content...


Our Website Development Notes

We hope this helps!

Articles // SQL // MySQL

Foreign Characters create symbols in PHP and MySQL

Wednesday, 17th August 2016
5,579 Reads
Applies to:
  • PHP 5.4
  • MySQL 5.1.4
  • Databases initially setup with collation 'latin1_swedish_ci'

Another article trying to help people display foreign characters on their website without the funny question marks in diamond symbols and how I solved it in my case.

My company has started using international country and region names which include foreign characters. When we copy and paste their content into our website, our webpages display a question mark inside a diamond shape instead of the foreign character.

How does it happen? Have I tried the other solutions on the web? I have tried adding the following to my headers:
  • This does nothing:
    1. // add these to the header: DOESNT WORK  
    2. mb_internal_encoding('UTF-8');  
    3. mb_http_output('UTF-8');  
    4. mb_http_input('UTF-8');
  • This does nothing:
    1. // add the following to your header  
    2. <meta charset='utf-8'>  
    3. <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  • This does nothing:
    1. // set the collation of the database and any text fields to 'utf8-general-ci'  
    2. ALTER DATABASE db_name CHARACTER SET latin1 COLLATE utf8-general-ci;  
    3. // set text fields to utf8  
    4. SET NAMES 'utf8';
  • This does nothing:
    1. // convert the foreign character using PHP to htmlentities  
    2. $my_description = htmlentities($my_description);  
    3. $my_description = htmlspecialchars($my_description);

Did I miss anything?
So none of the above worked. If there is another solution out there, I didn't find one that worked otherwise I'd have included it in this article.

The fix is a PHP one and has to do with versions of PHP and MySQL. As quoted by the PHP htmlspecialchars:
"As of PHP 5.4 they changed default encoding from "ISO-8859-1" to "UTF-8". So if you get null from htmlspecialchars or htmlentities..." Source: PHP Manual

My fix (in some cases):
  1. $my_description = html_entity_decode(htmlentities($my_description, ENT_COMPAT, 'ISO-8859-1', true), ENT_COMPAT, 'UTF-8');
A more updated version:
  1. $my_description = html_entity_decode(htmlentities($my_description, ENT_COMPAT, 'ISO-8859-15', true), ENT_COMPAT, 'UTF-8');
My other fix (for Croatian characters and Western languages):
  1. // insert after database connection and prior to database query (where $db_conn is your mysqli connection)  
  2. mysqli_set_charset($db_conn,"utf8");  
  4. // decode and display in HTML-safe  
  5. $my_description = html_entity_decode(htmlentities($my_description, ENT_COMPAT,'UTF-8', true), ENT_COMPAT, 'UTF-8')

Other things to consider:
  • Save all PHP files as UTF-8 and not ANSI
  • Ensure all text fields in the database are set to a UTF-8 charset (eg. utf8_general_ci)
  • Ensure headers are setting the charset to UTF-8
  • Add the charset attribute to posting HTML forms: accept-charset="utf-8"

The final fix in my case was due to me not specifying some extra parameters on the htmlentities or htmlspecialchars PHP functions. My database stored the foreign characters as they were so the fault was somewhere between PHP reading from MySQL to displaying the characters on the webpage.

Recent Comments

Gravatar for Art
Oracle: order by subquery missing right parenthesis
Hello, what if I would like to add rownum to that code, but in my case it's all is subquery? How can I do this ?

20 Jan

Gravatar for Robert
CharIndex Reverse - find occurrence starting from end of string in TSQL
Very helpful thanks! This worked for me as well, and is a little shorter. Not sure if there are drawbacks. SELECT RIGHT(@Haystack,CHARINDEX(@Delimiter,REVERSE(@Haystack))-LEN(@Delimiter))

29 Dec

Gravatar for Translation

27 Dec

Gravatar for Tibbe
JComments 2.3.0 with ReCaptcha in Joomla 2.5.x
Hi there, This looks like a great solution to get rid of spam comments. How to integrate this in Joomla 3.x? Step 2 I did in settings.xml, but no result in frontend. Kind Regards, Tibbe

16 Dec

Gravatar for sach|n
SSRS Repeat Headers in PDF Report
Yes its working for me..thnx Joel.

7 Oct