Currently:
final boolean needsQuoting = stringValue.indexOf(delimiter) != -1 //
|| foundQualifier //
|| stringValue.indexOf('\n') != -1;
Should be:
final boolean needsQuoting = foundQualifier && (stringValue.indexOf(delimiter) != 1 || stringValue.indexOf("\n") != 1);
Because:
If the qualifier is not found, then you cannot write the qualifier ever.