Comments:

• Mark Wilden · October 19th, 2006  · 4:41 pm  · Permalink

  The most challenging bugs I’ve had have always been ones where reproducing them was the most difficult step.

• John Lenton · October 19th, 2006  · 5:51 pm  · Permalink

  Why not detect that very image type (or have a list of unsupported image types), and convert it to something supported via ImageMagick?

• Katy · October 19th, 2006  · 6:09 pm  · Permalink

  I am glad I could be of help figuring out this most difficult and interesting bug.

  Again, thanks for you help, and the program has proven to be very useful thusfar.

• Steph · October 19th, 2006  · 10:02 pm  · Permalink

  Absolutely, often finding out how to reproduce a bug is much more challenging than fixing it!

  As for image types, that’s exactly what we’re doing. We accept JPG and GIF image types. However if it was encoded using in one specific mode (out of over 20 different ones) then it won’t work. And the key is that you can’t detect this with the built in libraries without rendering it (the image type says that it’s correct), which means you either have to get another third party library or understand the details of the image types.

  I don’t want really want to understand the details of how a JPG is encoded, LandlordMax’s core functionality is not as an image viewer, it just ads this ability.

  This doesn’t mean to say that we don’t care, it just means that we won’t provide support for every single image type. We will therefore limit the software to accepting only the major image types. I’d instead rather focus our energies on enhancing the software’s core features and performance.

  And yes Katy, you were very instrumental in figuring out this bug! Thank you.

• GUI Junkie · October 20th, 2006  · 3:02 am  · Permalink

  Why don’t you import the file to a ‘blob’ field? This way, your response time stays the same. Afterwards, you can move the image to the real field using a batch or a fork.

  For those of you that don’t program. A batch process is launched at predefined times, mostly when the server workload is lowest. A fork is a parallel process launched when importing the file.

• Leon Mergen · October 20th, 2006  · 5:50 am  · Permalink

  .. hmm and it isn’t possible to probe for the CMYK mode inside the file ?

  I have to do image processing as well (I need to send the correct HTTP content-type headers for an image without relying on its extension), and personally use probing as a method te determine the file type.. for example, if the binary data starts with 0xFF 0xD8 0xFF it is most likely a JPEG file.. I can only assume there must be some kind of header for CMYK mode too, whose information is “out there” somewhere.. 🙂

• A reader · October 20th, 2006  · 6:08 am  · Permalink

  Of course, the definitive test to know if you’re able to render an image is just to try to rendder it. But you can avoid this, and other corner cases, just by inspecting the image metadata that’s at the beginning of the JPG file.

  The problem is, the JPG format is very complex and you’ll be probably missing a few other exceptions. But at least you don’t need to render the image.

• John · October 20th, 2006  · 7:35 am  · Permalink

  Weird. I’ve come across that exact same problem within in the last few days but within my Delphi app. It had me scratching my head for a few hours too.

• Steph · October 20th, 2006  · 11:16 am  · Permalink

  This is great feedback! I’ve also received several personal emails with additional information. Thank you!

  Please find below my responses to your latest comments:

  GUI Junkie:

  Thank you for the suggesion, however just to let you know LandlordMax is a standalone desktop application (A Swing application). Becuase of this, people expect quick responses. We don’t have the options of batching it, pushing it to the middle of the night, etc. Honestly, if this was a web application, this issue would be a lot simpler and we’d have more options. But that’s life.

  Leon Mergen and A reader:

  You guys are right, I could manually go into the JPG’s metadata and figure out what’s going on. But again, the issue with this is that I’m not really interested in learning the details of how a JPG is encoded. Our primary focus is not displaying JPG’s, it’s about property management software. We’re only willing to support the primary image formats, not the exceptions. For example if we added email support into the software, we wouldn’t want to create another Sendmail (would you?). We might just want enough capabilities to support sending emails.

  And just like “A reader” suggested, the JPG format is very complex and we’d probably end up missing some other exceptions, or possibly make other mistakes. The cost to benefit ratio is just not there.

  My prefered solution would be to quickly and easily detect these types of JPG images and let the user know that the software won’t import the image because of that reason. Simple, quick, and it will satisfy 99.999% of our customers. Remember they’re not buying LandlordMax to view their pictures, their buying it to manage their real estate properties, and storing pictures and their logo is only a small part of that.

  John:

  It’s interesting that Delphi also has this issue. I’ve also heard since putting out the article that other programming languages have this same issue with that JPG mode. As well, some libraries don’t like the Photoshop JPG preview mode either. Just a heads for those of you starting out with this issue.

• Barney · October 20th, 2006  · 12:18 pm  · Permalink

  Maybe a solution is to have this as a prominent FAQ item: “If you get this error … then you may be trying to save an RGB file in CMYK mode (which is mainly used for printing). Try saving it in ‘Image -> Mode -> RGB Color’ mode (if using Photoshop) and then loading it into our program.”

• Jacob Proffitt · October 20th, 2006  · 12:54 pm  · Permalink

  One quick solution that is likely to satisfy both development time constraints and user interaction concerns might be to add an error capture around where this error happens and add a short summary of why the error is likely happening and perhaps a link to a help file or web page that explains how save in RGB. The help file/web page should have a very short summary and a simple solution using software your clients likely have on hand (I’m assuming MS Paint can fix this pretty easily). Add a link to more detailed information, either here or the resources you found, and you’re golden.

  Your clients aren’t vested in using CYMK over RGB, and they have control over the file anyway (having been the ones to add it to the database). Your clients can chose the level of detail they want about the error. That way your clients are trained in the issue, it becomes non-business stopping, and the burden of working around this issue is redistributed to the benefit of all.

• gman · October 20th, 2006  · 2:39 pm  · Permalink

  You might look into libjpeg. It’s FreeBSDed. Maybe it handles CYMK. It’s in C but you could probably make something you could call out to to convert the image. Just a suggestion.

• John Lenton · October 20th, 2006  · 6:01 pm  · Permalink

  Sorry, maybe I wasn’t clear about the “image types” thing, or maybe I’m being obtuse. There is a library called “ImageMagick” (https://imagemagick.org) that you can use, that will not only tell you anything you want to know about most any image you can feed it, it can also convert between any two formats, including converting between jpeg cmyk and jpeg rgb. I know, because I tried.

  Alas, the java bindings are not complete. That might be a problem for you, I guess…

• AndrewT · October 20th, 2006  · 6:40 pm  · Permalink

  The flag for the color mode is hidden somewhere in the file, so it is only a matter of time/effort to find it once you get the file uploaded:

  a) spend another day to learn the jpeg specs;

  b) grab some open source code that works with the jpegs and cut this time by half;

  c) find someone who already knows jpegs well to write a simple check code for you (i.e. put in some of your money instead of your time).

  If you are unsure of the quality of the new check (e.g. scared of false positived), use it as a pre-step for the rendering procedure: this way, only “suspicious” images will have to be rendered, with way less overhead.

• Bill Mill · October 20th, 2006  · 11:18 pm  · Permalink

  Andrew,

  Have you read the JPEG spec? Or, I should say, tried to? Because I have. Here’s a link: https://tools.ietf.org/html/rfc1341 .

  It is, by far, the most difficult to understand non-WS-* spec that I have ever come across. (And I’ve probably read more than my share of specs). All I wanted to do at one point was to write a program to add a comment to the exif metadata of a jpeg file, and it proved to be darn near impossible.

  He’s looking at more than a day to write that code, and he’s likely to never actually have code that is correct. Normally I’d agree with you, but in this case, the spec is so horribly broken that it’s not worth the pain.

  I encourage a bit of reading. Try reading that spec, or here’s Mozilla’s jpeg renderer.
  You might be surprised at how big a problem this is.

  -Bill Mill
  bill.mill at gmail.com

• Bill Mill · October 20th, 2006  · 11:31 pm  · Permalink

  Andrew,

  Sorry, I gave an incorrect reference above: My memory was off, I was trying to read the exif spec at https://exif.org/specifications.html . The JPEG spec is ISO 10918-1, and is not available online. (You can buy it at https://www.iso.org/iso/en/CatalogueDetailPage.CatalogueDetail?CSNUMBER=18902 if you’re so inclined.)

  It’s still wildly complex, and I still welcome you to tell me, from the Mozilla code, some easy way of figuring out the CMYK-ness of a file, for some large subset of JPEG images.

  Sorry for the factual errors.

  -Bill Mill
  bill.mill at gmail.com

• GUI Junkie · October 21st, 2006  · 4:55 am  · Permalink

  Ok, no batch.
  What about parallel processing? Java does know how to fork.

• Steph · October 21st, 2006  · 4:10 pm  · Permalink

  Hi everyone,

  This is great feedback! The comments you’ve put (and links) really help to put into perspective just how big an issue this is!!!

  My responses:

  Barney and Jacob:

  This is probably close to what I’ll end up doing. The only issue is that if I don’t pre-render it, I won’t know about the error until after it’s trying to display it. Now in a desktop application, I can either put an “X” type image (or something to show it failed) and allow the user to continue with that image (or delete it). That or I attempt to pre-render it… I’m strongly leaning away from checking the image file or converting it. So is the performance hit worth it to pre-check? I’m starting to think no since it’s a small percentage of people who will encounter this issue and it’s easily fixable by the client with the error message and the “X” image replacement (with some good documentation, etc.)…

  GUI Junkie:

  I’ve developed both server and desktop applications, and I’m not sure where you’re experience lies, so please take this with the utmost respect. In GUI applications, you don’t want your application firing off in the background a process to later pop-up an error window that can affect your current work. If you look, even a virus scan locks you out of many tasks while it’s performing the scan (you can’t change the scan parameters in the middle of a scan). The issue is that if I fork off a thread, the user saves the building say, then what happens if there’s an error in the imported images? What if they change the values? It’s better to put a modal progress bar while it’s loading. Of course the faster you can make things happen, the happier your user is. This is generally what desktop users are use to.

  In a server environment, you have a lot more leeway in what you can do. People’s expectations are also very different on how a web application and a desktop application behave. For example, in a desktop application, anything that takes more than 2 seconds makes the user think your program has frozen or is extremely slow. In a web application, most web pages take more than 2 seconds to load and transmit! Think about Amazon.com, it takes many seconds to render almost any page. This would be unacceptable in a desktop application…

  User expectations also go the other way too. In a web application, you can’t lock out a user from making changes from one screen to another (one web browser to another). You need to manually do those checks. In a desktop application, all you need to do is make the GUI component modal (or possibly add a progress bar if it’s long enough).

  Different presentations, different rules. This is also why it’s very difficult for many people to move from a web application to a desktop application. Few people can, the rules are different, not to mention the libraries 😉 Also, until recently, most web application were action driven, whereas desktop applications are event driven. I’m slowly seeing this change with new Java frameworks like JSF, which is great!

  The others:

  I’m strongly leaning against manually converting the image, or manually checking for the flags in the meta-data. I think Bill Mill said it best, just look at the specs! We’re not an imagine processing company, nor do we want to be, so why spend an innordinate amount of time on it. And after look at how Mozilla renders it, and knowing that the very same image that I was testing with failed in Mozilla, I know when it’s time to look elsewhere. Yes it failed in the Mozilla code!!!

  As well, just to let you know, we’re using the main Java librairies to render it. This is not our code, this is a core language library which has this known issue in it. Mozilla (FireFox) also doesn’t have it fully implemented. Internet Explorer also had some issues with it. Looking at the scale of this JPG issue, and that only those software specialized in images have it working (and not all of them, I found some open source imaging solutions that also described this very same issue), I don’t think we should be taking it head-on. The cost to benefit is nowhere near the return on investment!!!

  Thanks Bill, you showed just how complex this issue is!

• GUI Junkie · October 25th, 2006  · 7:54 am  · Permalink

  Let me quote: “Right now I’m personally leaning towards doing an extra validation and trying to invisibly render it from the file directly before accepting it.”

  Now, your objection to yourself is “I’m not satisfied with it yet as it will have a lot of performance overhead” and that is something you might consider doing with a fork.

  For example. When accepting an image, you
  1) show the reception of the image(s) and it’s status.
  2) fork the validation process, this process can update the status of the image.

  This way, you get what you want. Speed & Validation.

  I don’t know whether you can reduce the thread priority of the fork so it doesn’t slow down your desktop app.

• Mark · October 26th, 2006  · 9:20 am  · Permalink

  I agree that the JPG format is highly complicated. One thing that is not complicated is the header. It is very easy to read the header and with the header, you can tell if an image is CYMK or not. Here is some code you can use.(Needs a little cleaning up):

  import java.awt.Graphics;
  import java.awt.Image;
  import java.io.ByteArrayInputStream;
  import java.io.ByteArrayOutputStream;
  import java.io.FileInputStream;
  import java.io.InputStream;

  import javax.imageio.ImageIO;
  import javax.swing.JFrame;

  public class Test extends JFrame {
  Image im;
  public void loadImage(InputStream in) throws Exception {
  ByteArrayOutputStream bout = new ByteArrayOutputStream();
  byte buffer[] = new byte[8192];
  int read = -1;
  do {
  read = in.read(buffer);
  if(read != -1) {
  bout.write(buffer, 0, read);
  }
  } while(read != -1);
  byte imageData[] = bout.toByteArray();
  int startOfHeader = -1;
  for(int i=0;i

• Mark · October 26th, 2006  · 9:21 am  · Permalink

  rest of code:

  for(int i=0;i

• Mark · October 26th, 2006  · 9:22 am  · Permalink

  Sorry, forgot about entities:
  import java.awt.Graphics;
  import java.awt.Image;
  import java.io.ByteArrayInputStream;
  import java.io.ByteArrayOutputStream;
  import java.io.FileInputStream;
  import java.io.InputStream;

  import javax.imageio.ImageIO;
  import javax.swing.JFrame;

  public class Test extends JFrame {
  Image im;
  public void loadImage(InputStream in) throws Exception {
  ByteArrayOutputStream bout = new ByteArrayOutputStream();
  byte buffer[] = new byte[8192];
  int read = -1;
  do {
  read = in.read(buffer);
  if(read != -1) {
  bout.write(buffer, 0, read);
  }
  } while(read != -1);
  byte imageData[] = bout.toByteArray();
  int startOfHeader = -1;
  for(int i=0;i<imageData.length – 1;i++) {
  if((imageData[i] & 0x000000ff) == 0xff && (imageData[i + 1] & 0x000000ff) == 0xc0) {
  startOfHeader = i;
  break;
  }
  }
  //if image type = 4 CYMK mode is used
  int imageType = bout.toByteArray()[startOfHeader + 9];
  System.out.println(“type = ” + imageType);
  im = ImageIO.read(new ByteArrayInputStream(bout.toByteArray()));
  setSize(im.getWidth(null), im.getHeight(null));
  }
  public void paint(Graphics g) {
  g.drawImage(im, 0, 0, this);
  }
  public static void main(String args[]) {
  Test t = new Test();
  try {
  t.loadImage(new FileInputStream(“any_image.jpg”));
  t.setVisible(true);
  t.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
  }
  catch(Exception e) {
  e.printStackTrace();
  }
  }
  }

• Steph · October 26th, 2006  · 1:19 pm  · Permalink

  Hi Mark,

  Thanks for posting all the code!

  Do you know how thoroughly this code has been tested? The reason I ask is because from what I’ve heard the way the meta data is encoded can be slightly different (the specs are a little ambigiuous at times). From I see in your code, you’re expecting the CMYK flag (aka. image type flag) to appear at a specific location within the header block (9 bytes from the start of the header)…

  From what I understand, Photoshop for example can also throw in a thumbnail within the image which can skew the whole thing too? Do you know anything about this?

• Mark · October 26th, 2006  · 2:38 pm  · Permalink

  Thumbnails or any other anomolies should not cause a problem. From the specs I read, there are certain guaruntees. All information is divided into segments. A segment starts with the the byte 0xFF and 0xFF will only be used to denote the start of a segment. The next byte after 0xFF says what type of segment you have. For the header segment, the bytes 0xFF 0xC0 will appear in that order. The combination of 0xFF followed by 0xC0 will only appear once in any JPEG file. Also the header size is fixed, so the 9th position contains the image type flag always.(Counting 0xFF as position 0.)

  The code is not thoroughly tested. I just wrote it.

  Here is the header part of the JPEG spec.

  SOF0: Start Of Frame 0:
  ~~~~~~~~~~~~~~~~~~~~~~~

  – $ff, $c0 (SOF0)
  – length (high byte, low byte), 8+components*3
  – data precision (1 byte) in bits/sample, usually 8 (12 and 16 not
  supported by most software)
  – image height (2 bytes, Hi-Lo), must be >0 if DNL not supported
  – image width (2 bytes, Hi-Lo), must be >0 if DNL not supported
  – number of components (1 byte), usually 1 = grey scaled, 3 = color YCbCr
  or YIQ, 4 = color CMYK)
  – for each component: 3 bytes
  – component id (1 = Y, 2 = Cb, 3 = Cr, 4 = I, 5 = Q)
  – sampling factors (bit 0-3 vert., 4-7 hor.)
  – quantization table number

  Remarks:
  – JFIF uses either 1 component (Y, greyscaled) or 3 components (YCbCr,
  sometimes called YUV, colour).

  The spec is on http://www.wotsit.org if you want to look at it fully. The segment definitions are near the end.

• Steph · October 26th, 2006  · 2:44 pm  · Permalink

  Hi Mark,

  Please take this with the utmost of respect. If it was this simple (I’m not implying that getting to this snippet didn’t take a lot of time and understanding) where one could just write out the code (you mention you just wrote it and didn’t thoroughly test it), then it makes be believe that Mozilla and Internet Explorer should have been able to get it right with all their massive resources? So the question is why didn’t they?

  By this reasoning, I’m going to assume there’s more than meets the eye…

• Mark · October 27th, 2006  · 7:24 am  · Permalink

  I understand your skeptisism. This code does not allow you to display it. It only allows you to detect if you can. The rest of the spec talks about Huffman encoding and the like, which would take a lot of time to understand.(I have not taken that time.) Don’t let your skeptisism make you automaticaly dismiss a solution that could be correct. The code snippet is so small that it could easily be inserted to try it out and removed later if you still don’t trust it.

  Also consider your blog related to this where you explain that it is not cost effective for you to go with a developer oriented solution, but need to go more with a business oriented solution. Consider that Microsoft and the Firefox crew have huge budgets and may be only considering the developer oriented solution. As you get bigger sometimes you can get blinded to the simple solution.

  Good luck.

• Mark · October 27th, 2006  · 11:02 am  · Permalink

  Damn, spelled skepticism wrong. There goes my credibility. 🙂

• Steph · October 27th, 2006  · 4:28 pm  · Permalink

  Hi Mark,

  You’re right, I skipped over the part that the code “only” lets you detect it, nothing more. You’re absolutely right, my apologees.

  One thing though that I’ve been thinking of since then, no matter what kinds of checks we do, we still have to implement the code in case there is an error. For example, maybe there’s another format that will cause a similar issue. I’ve read something about the Photoshop thumbnail preview mode causing issues with many image renderers too.

  So the question then becomes, do we just implement one solution or two? Seeing your code, it’s a fairly simple implementation, so it makes me want to implement such a check. But then I start to think about testing it. I’ve only come accross one image that has this issue (of course I can create others). But now how do I test for false positives?

  Testing for false positives can become very complex, there are many image modes. Since we also have to implement the other solution because of this, is it worth going through this extra effort?

  That’s a hard question. Right now I’m leaning towards no. Not because the solution is difficult, you provided a great succint solution (thank you!), but testing it is very difficult. We need to test images that have this issue, and all other image formats to make sure we’re not preventing a valid mode. This false positive testing is starting to scare me in terms of costs…

  And if we look at the gain, since we have to implement the other solution that deals with all cases, what’s the gain of this additional check? It’s an initial performance gain when first importing the images (so we never have to try to render them). This is why I’m going to go with just the simple solution of catching the errors when rendering at this time. Yes it’s more expensive to implement, but we have to implement it anyways, and it’s much cheaper to test.

  And yes, you’re absolutely right, sometimes the obvious simple solution is the hardest to see! I know I’ve been guilty of that one more than once 🙂

  These days, I always try to find some type of justification for what we add into LandlordMax. When you’re smaller every penny counts that much more! It’s also easy to fall into the trap of not fixing bugs, but I’m very opposed to this because I’m strong believer of design debt. However, if the bug fix is very large, the frequency of occurence very low, and the spread non-existant (only affects this, it does not increase design debt), then sometimes you need to look at the alternatives, as was the case with this bug.

  To give you an idea of some bug fixes that we can’t justify with just the model I described above is performance enhancements. We’ve done some major performance enhancements with the upcoming major release (as well as with version 2.12). We spent a lot of time and effort, almost 1/2 of it was performance based!

  Will this automatically convert to sales? Maybe, maybe not. However if we hadn’t made these enhancements we would be acquiring a lot of design debt which would have hit us later. Maybe not tomorow, but soon enough. When it comes to performance you’re only as strong (fast) as the weakest (slowest) link in your chain.

• Mark · October 30th, 2006  · 8:19 am  · Permalink

  Since there are anly 3 colour modes for JPEG, I don’t think false positives would be a problem. But I can see a problem with thumbnails. They are not declared in the header, so you would have to find them separately. A tough chore.

  Good luck with your solution.

• Steph · October 30th, 2006  · 1:57 pm  · Permalink

  Hi Mark,

  I didn’t know that… From my initial impressions searching on Google I was pretty sure there were 20 or so different encodings. To be quite honest, I’m very ignorant as to the details of the JPG encodings, and I don’t really have an interest in knowing them, I’m just too busy on other higher priority items.

  It’s s very interesting to know about the thumbnail! Thank you for sharing that information. Therefore from what I gather they aren’t part of the spec if their not in the header, something that was added after by Photoshop and similar software?

  Thanks for the information, it’s very appreciated. In regards to the solution we’re going to implement, you’re comments and suggestions are still making me ponder which path to take… Sometimes I find that if I just sit on the options for a while, let it work in the back of my mind, it easier for me. At this point I haven’t decided yet.

Write a reply: