OCR and Data Extraction
A big part of any OCR solution is the process of data capture and extraction. Most document capture applications provide the ability to process the converted text and provide the extraction of expressions from the text. So how can this help? Well, the ability to parse the OCR text provides automation, and allows you to populate fields based on what you find.
An example might be a form that has DOB: 1/2/1968
You want to extract everything to the right of DOB: from a document. You can do this with an ADE engine.
Tuesday, March 30, 2010
Saturday, March 6, 2010
OCR and the Right Settings
What DPI should be set for optimal OCR Accuracy?
So. I get this question all the time and decided it might be good to post about it. What is the best DPI setting for Optical Character Recognition (OCR)?
I have been at clients that erroneously believe the higher the DPI, the beeter the results, and feel pain whenever I see an OCR Scanner set beyond 300 DPI, and some even at 600 DPI!! Holy cow, how do you handle those file sizes?
The fact remains that almost all OCR engines on the market are tuned and optimized for 300DPI for optimal conversion and recognition. Going beyond this will provide no better results, and significantly increase your file size exponentially. Most Document Capture companies provide image processing prioer to OCR that will allow you to scan at 200 DPI, with fairly consistent results.
So. I get this question all the time and decided it might be good to post about it. What is the best DPI setting for Optical Character Recognition (OCR)?
I have been at clients that erroneously believe the higher the DPI, the beeter the results, and feel pain whenever I see an OCR Scanner set beyond 300 DPI, and some even at 600 DPI!! Holy cow, how do you handle those file sizes?
The fact remains that almost all OCR engines on the market are tuned and optimized for 300DPI for optimal conversion and recognition. Going beyond this will provide no better results, and significantly increase your file size exponentially. Most Document Capture companies provide image processing prioer to OCR that will allow you to scan at 200 DPI, with fairly consistent results.
Wednesday, February 24, 2010
Tuesday, February 23, 2010
OCR Software and Character Correction
Optical Character Recognition and Character Correction
So what is character correction when associated with OCR? The OCR process provides the recognition and conversion of images to text, and in this process, there can be many characters that can be misidentified throughout the conversion process. Typically, document capture applications provide the ability to identify commonly misinterpreted characters through a table of correction mappings. So lets say a particular zone OCR field was designated as numbers only, and the engine interpreted an "l" for a "1" (that is an l for a one). The correction piece of the recognition engine can provide logic to the OCR process, and make sure the text is properly interpreted. This can be really important, especially in SharePoint OCR environments where you need searchable PDFs in SharePoint.
This is just one of many ways to improve accuracy, but note you will need the right kind of OCR application that allows this feature to be enabled.
So what is character correction when associated with OCR? The OCR process provides the recognition and conversion of images to text, and in this process, there can be many characters that can be misidentified throughout the conversion process. Typically, document capture applications provide the ability to identify commonly misinterpreted characters through a table of correction mappings. So lets say a particular zone OCR field was designated as numbers only, and the engine interpreted an "l" for a "1" (that is an l for a one). The correction piece of the recognition engine can provide logic to the OCR process, and make sure the text is properly interpreted. This can be really important, especially in SharePoint OCR environments where you need searchable PDFs in SharePoint.
This is just one of many ways to improve accuracy, but note you will need the right kind of OCR application that allows this feature to be enabled.
Friday, February 19, 2010
Are index fileds really necessary when you have the full text OCR?
Full text OCR
Ah, the old debate, do I just perform optical character recognition on all my scanned documents, make them searchable OCR PDFs, and rely on the OCR to retrieve documents? Why use index fields when I already have all the converted text?
Index fields, or performing the indexing process, provides structured data about the documents. This data can be utilized, especially when using document capture software, to link into columns and index fields in your document management system. Index fields provide faster retrieval, especially if you want to be able to retrieve through specifying several criteria. Relying on OCR, or the recognized text can get you in trouble. First of all, you are assuming that the document will alwyas have recognized text, and that all the items that you are searching for are in the text. Secondly, depdning on the type of OCR format you have, you may have to just find the document, and then open and parse what you are looking for. This can also lead to false positives in retrieval if many documents have the same terms in their OCR text.
Ah, the old debate, do I just perform optical character recognition on all my scanned documents, make them searchable OCR PDFs, and rely on the OCR to retrieve documents? Why use index fields when I already have all the converted text?
Index fields, or performing the indexing process, provides structured data about the documents. This data can be utilized, especially when using document capture software, to link into columns and index fields in your document management system. Index fields provide faster retrieval, especially if you want to be able to retrieve through specifying several criteria. Relying on OCR, or the recognized text can get you in trouble. First of all, you are assuming that the document will alwyas have recognized text, and that all the items that you are searching for are in the text. Secondly, depdning on the type of OCR format you have, you may have to just find the document, and then open and parse what you are looking for. This can also lead to false positives in retrieval if many documents have the same terms in their OCR text.
Tuesday, February 16, 2010
Zone OCR and Accuracy within Recognition Zones
Zone OCR Accuracy
So when doing zone OCR , or Optical Character Recognition on a portion of a page, what features do I need to ensure I have the best possible accuracy. List below:
So when doing zone OCR , or Optical Character Recognition on a portion of a page, what features do I need to ensure I have the best possible accuracy. List below:
- Utilize a document capture application that provides some type of page registration. The problem with using zone OCR is that most engines utilize a set template of coordinates on the page, and just repeat this "zone" on each page. If the scanner is off, or the page skewed, you can have erroneous readings. Page registration gives the recognition engine the ability to anchor a page feature, always referencing the zone from the set coordinates of the feature.
- Utilize a scanning application that provides the ability to perform image processing on the zone prior to running Optical Character Recognition . Removing lines, deshading, despeckling can provide a cleaner zone, and thus improve overall accuracy.
- Some advanced capture applications provide the ability to filter zones based on character sets. This allows you to interpret the characters within a zone as say, all numbers, or perhaps a date, which provides the engine a more narrower character set for the whole recognition process. iCapture for example, not only allows character set mapping to zone ocr templates, but also provides auto-correction for the most commonly misinterpreted characters.
- Finally, and highly recommended for the highest level of accuracy, is the ability to set a character matching filter for a zone. This technology, sometimes called ADE, provides the ability to utilize regular expressions to ensure a match, and lets you over draw the recognition area / zone and filter to your liking.
Saturday, February 13, 2010
Why use OCR Software to perform full text conversion of images?
OCR Software
When we scan documents, they are just images, pictures of our paper. For many organizations, this scanned image is exactly what they need, and a little index information about the document is sufficient to provide them with retrieval capability.
So why take the time and spend the money to utilize OCR Software to convert the scanned document to a searchable format? Below are some reasons to always perform full text OCR of scanned documents:
When we scan documents, they are just images, pictures of our paper. For many organizations, this scanned image is exactly what they need, and a little index information about the document is sufficient to provide them with retrieval capability.
So why take the time and spend the money to utilize OCR Software to convert the scanned document to a searchable format? Below are some reasons to always perform full text OCR of scanned documents:
- Always provide every means possible for retrieval. Just using index fields to search for scanned documents may seem like a fantastic idea, but what if the document is misidentified? Or the indexer enters incorrect information? Performing a full text OCR of the document can provide an insurance policy that a document can always be found through full text search.
- Document Capture software today provides fast reliable OCR. Most capture software on the market provides the ability to automatically convert the documents to searchable format for a small expense. Some of the engines on the market can do the conversion at 100+ pages per minute, so there is really not much time wasted in the OCR conversion / recognition process.
- OCR to PDF for a format that contains both image and text in one container. Adobe provides the PDF image with hidden text option to give you a seachable file format that contains a pristine image.
- Plan for the worst case. Audits...legal issues...sometimes you need to search beyond the index fields, and full text can give you the ability to find the needle in the haystack.
Tuesday, February 9, 2010
OCR Software - Distributed vs. Centralized
OCR Software - Distributed vs. Centralized
Ah, the centralized versus distributed question...it is one that is continually asked in the scanning, capture and document capture space. Most associate OCR Software with familiar desktop applications like eCopy Desktop, OmniPage, PaperPort, etc. These provide, in a way, distribution of the overall OCR process to end users.
There are applications on the market that can provide centralized and controlled OCR capabilities, through either a server or a workstation deployment. One example is PSI:Capture from PSIGEN, and advanced document capture application, that allows centralized OCR processing. Why would you want to do this? Well, in most cases, this type of OCR deplyment model is utilized in conjunction with a document capture system, for centralized capture, indexing, QA, OCR and migration to a centralized DM / ECM system. Typically, these systems give a broad and expansive feature set, providing all different types of OCR functionality.
Ah, the centralized versus distributed question...it is one that is continually asked in the scanning, capture and document capture space. Most associate OCR Software with familiar desktop applications like eCopy Desktop, OmniPage, PaperPort, etc. These provide, in a way, distribution of the overall OCR process to end users.
There are applications on the market that can provide centralized and controlled OCR capabilities, through either a server or a workstation deployment. One example is PSI:Capture from PSIGEN, and advanced document capture application, that allows centralized OCR processing. Why would you want to do this? Well, in most cases, this type of OCR deplyment model is utilized in conjunction with a document capture system, for centralized capture, indexing, QA, OCR and migration to a centralized DM / ECM system. Typically, these systems give a broad and expansive feature set, providing all different types of OCR functionality.
Tuesday, February 2, 2010
How fast is OCR Software? OCR Performance Testing
OCR Performance Testing
So which Desktop Optical Character Recognition Software is the fastest? Has the best overall performance when converting images to Word? When converting images to PDF?
I ran some testing with 4 basic desktop OCR applications to see which would have the fastest conversion times. The OCR applications are:
-eCopy Desktop (Uses the ReadIRIS OCR Engine)
-Adobe 8
-Paperport 11
-OmniPage 15
I ran all the tests on a 9 month old laptop, with a Dual Core 2 GHz processor, and 2 GB of memory. I utilized all the "out of the box" settings on the apps, with no performance tuning of settings, and I timed the speed of the applications to convert a 100 page TIFF image to Word and to Adobe Image and Text PDF.
Results of the OCR Speed Test in minutes and seconds(Word/PDF):
eCopy Desktop 4:25/2:58
Adobe 8 3:54/3:22
OmniPage 15 2:16/2:16*
PaperPort 11 2:35**
*With OmniPage you run the conversion process and then save to your preferred format.
**PaperPort just had text conversion capabilities.
I have to note that the eCopy Desktop test can be misleading in that it performs auto-orientation on all the pages before performing OCR. Also note that when evaluating an OCR application, speed is not the only factor. You need to decide up front whether you want speed, accuracy, both, or want to focus on formatting. I will write another article on formatting and which application is best in the near future.
So which Desktop Optical Character Recognition Software is the fastest? Has the best overall performance when converting images to Word? When converting images to PDF?
I ran some testing with 4 basic desktop OCR applications to see which would have the fastest conversion times. The OCR applications are:
-eCopy Desktop (Uses the ReadIRIS OCR Engine)
-Adobe 8
-Paperport 11
-OmniPage 15
I ran all the tests on a 9 month old laptop, with a Dual Core 2 GHz processor, and 2 GB of memory. I utilized all the "out of the box" settings on the apps, with no performance tuning of settings, and I timed the speed of the applications to convert a 100 page TIFF image to Word and to Adobe Image and Text PDF.
Results of the OCR Speed Test in minutes and seconds(Word/PDF):
eCopy Desktop 4:25/2:58
Adobe 8 3:54/3:22
OmniPage 15 2:16/2:16*
PaperPort 11 2:35**
*With OmniPage you run the conversion process and then save to your preferred format.
**PaperPort just had text conversion capabilities.
I have to note that the eCopy Desktop test can be misleading in that it performs auto-orientation on all the pages before performing OCR. Also note that when evaluating an OCR application, speed is not the only factor. You need to decide up front whether you want speed, accuracy, both, or want to focus on formatting. I will write another article on formatting and which application is best in the near future.
Saturday, January 30, 2010
OCR Software versus Document Capture Software
OCR Software versus Document Capture Software
So all OCR Software companies provide the ability to convert scanned files into text or searchable PDFs via the Optical Character Recognition process, but how do I capture/scan the images so the applications can do their conversion?
This is an interesting question. Let's talk about Document Capture first. This type of application is built from the ground up to scan/capture documents at a high rate of speed, provide the means to collect information about the documents through a number of means, and then export the document/data to a back end repository. All document capture companies provide all types of OCR options, and usually OEM their OCR, ICR, OMR components from the major OCR application companies, like: ABBYY, OpenText, Nuance, ReadSoft, etc. Most of these companies have diversified their offering to include document capture, but their offerings far way short on the capture side in my opinion...they are OCR companies.
The real goal here is to get the best OCR results possible through a powerful OCR engine, and also minimize your time required to scan and process through the best document capture software. So, if you are looking to do high volume OCR processing, I highly recommend choosing a capture application that utilizes your OCR engine of choice to get the best of both worlds. I will write more on this topic in upcoming posts. If you want some guidance on How to pick the right OCR Software, click on the link text.
So all OCR Software companies provide the ability to convert scanned files into text or searchable PDFs via the Optical Character Recognition process, but how do I capture/scan the images so the applications can do their conversion?
This is an interesting question. Let's talk about Document Capture first. This type of application is built from the ground up to scan/capture documents at a high rate of speed, provide the means to collect information about the documents through a number of means, and then export the document/data to a back end repository. All document capture companies provide all types of OCR options, and usually OEM their OCR, ICR, OMR components from the major OCR application companies, like: ABBYY, OpenText, Nuance, ReadSoft, etc. Most of these companies have diversified their offering to include document capture, but their offerings far way short on the capture side in my opinion...they are OCR companies.
The real goal here is to get the best OCR results possible through a powerful OCR engine, and also minimize your time required to scan and process through the best document capture software. So, if you are looking to do high volume OCR processing, I highly recommend choosing a capture application that utilizes your OCR engine of choice to get the best of both worlds. I will write more on this topic in upcoming posts. If you want some guidance on How to pick the right OCR Software, click on the link text.
Tuesday, January 26, 2010
Microsoft SharePoint and OCR
Microsoft SharePoint and OCR
Scanning with Microsoft SharePoint is an interesting endeavor, and typically the main reason for this undertaking is to have a searchable body of information. So what type of Optical Character Recognition (OCR) Software can be utilized with SharePoint? First of all, all the same rules apply in picking the right recognition software to do the conversion from image to text, as outlined in "How do I pick the right OCR Software?". You need to evaluate what you are trying to accomplish and look at your business process and workflows to get a good idea of how to initiate the conversion process. Below are some key questions when evaluating a SharePoint OCR Solution.
Are your paper images scanned en masse, through a centralized capture process?
If this is the case, you would typically do all of your OCR processing and recognition in front end document capture software. These application provide the fastest OCR engines, and their recognition processing time can be anywhere from 100-600 pages per minute, depending on the types of pages you are scanning.
Do you utilize MFPs / Copiers to scan document to sharepoint?
Most companies are trying to leverage their investment in their copier hardware to provide end users a great scanning and capture onramp to SharePoint. In this case, you typically want an OCR application that can provide recognition on the fly, and do the conversion process behind the scenes. Their are many MFP integrated applications on the market that can provide the OCR engine: iCapture, NSI AutoStore, eCopy to name a few.
Do the end users compile, combine and work with documents at their desktops?
In environments where end users are constantly working in their documents, and need desktop scanning access, typically and OCR Desktop application can be the best solution. These applications can put the control of the conversion process in the end user's hands, and can provide them OCR capability at the click of the mouse. Some apps in this class are eCopy Paperworks, PaperPort and OmniPage.
Do you want to SharePoint OCR PDFs?
Knowing what format and how you want to search can be critical, and having OCR PDFs in SharePoint can allow for full text search.
All of the OCR Solutions on this page focus on doing the process before the documents hit SharePoint. I will write an article later on solutions that can OCR documents within SharePoint Libraries later.
Scanning with Microsoft SharePoint is an interesting endeavor, and typically the main reason for this undertaking is to have a searchable body of information. So what type of Optical Character Recognition (OCR) Software can be utilized with SharePoint? First of all, all the same rules apply in picking the right recognition software to do the conversion from image to text, as outlined in "How do I pick the right OCR Software?". You need to evaluate what you are trying to accomplish and look at your business process and workflows to get a good idea of how to initiate the conversion process. Below are some key questions when evaluating a SharePoint OCR Solution.
Are your paper images scanned en masse, through a centralized capture process?
If this is the case, you would typically do all of your OCR processing and recognition in front end document capture software. These application provide the fastest OCR engines, and their recognition processing time can be anywhere from 100-600 pages per minute, depending on the types of pages you are scanning.
Do you utilize MFPs / Copiers to scan document to sharepoint?
Most companies are trying to leverage their investment in their copier hardware to provide end users a great scanning and capture onramp to SharePoint. In this case, you typically want an OCR application that can provide recognition on the fly, and do the conversion process behind the scenes. Their are many MFP integrated applications on the market that can provide the OCR engine: iCapture, NSI AutoStore, eCopy to name a few.
Do the end users compile, combine and work with documents at their desktops?
In environments where end users are constantly working in their documents, and need desktop scanning access, typically and OCR Desktop application can be the best solution. These applications can put the control of the conversion process in the end user's hands, and can provide them OCR capability at the click of the mouse. Some apps in this class are eCopy Paperworks, PaperPort and OmniPage.
Do you want to SharePoint OCR PDFs?
Knowing what format and how you want to search can be critical, and having OCR PDFs in SharePoint can allow for full text search.
All of the OCR Solutions on this page focus on doing the process before the documents hit SharePoint. I will write an article later on solutions that can OCR documents within SharePoint Libraries later.
Saturday, January 23, 2010
What is OCR, ICR and OMR?
What is OCR, ICR and OMR?
In the area of text conversion, there is often confusion on the acronyms that surround the industry, and what each one designates. Below are some quick overviews of each of the recognition technologies, and what they accomplish:
Optical Character Recognition (OCR) Software
OCR Software takes images, and converts them to searchable text. The output can be a plain text file, or the industry standard today is an image with hidden text PDF. OCR can also be utilized to extract data from scanned images, providing a means to either harvest information, or create index fields for later search. OCR Software Definition
Intelligent Character Recognition (ICR) Software
ICR Software provides the ability to recognize handwritten, or hand printed text. This process can be extrememly accurate when the printed text is bound by boxes, or combed form fields. Hanwriting is a little more complex, and typically requires many samples to be accurate. ICR Software Definition
Optical Mark Recognition (OMR) Software
OMR Software, somtimes called "mark sesnse", provides the ability to read checked boxes on forms or documents. The software senses the difference between an unmarked and marked box using a baseline reading, and then allows the recognition to take place.
Many manufacturers combine all 3 into a single recognition engine that provides powerful analysis capabilities for scanned documents and forms. OMR Software Definition
In the area of text conversion, there is often confusion on the acronyms that surround the industry, and what each one designates. Below are some quick overviews of each of the recognition technologies, and what they accomplish:
Optical Character Recognition (OCR) Software
OCR Software takes images, and converts them to searchable text. The output can be a plain text file, or the industry standard today is an image with hidden text PDF. OCR can also be utilized to extract data from scanned images, providing a means to either harvest information, or create index fields for later search. OCR Software Definition
Intelligent Character Recognition (ICR) Software
ICR Software provides the ability to recognize handwritten, or hand printed text. This process can be extrememly accurate when the printed text is bound by boxes, or combed form fields. Hanwriting is a little more complex, and typically requires many samples to be accurate. ICR Software Definition
Optical Mark Recognition (OMR) Software
OMR Software, somtimes called "mark sesnse", provides the ability to read checked boxes on forms or documents. The software senses the difference between an unmarked and marked box using a baseline reading, and then allows the recognition to take place.
Many manufacturers combine all 3 into a single recognition engine that provides powerful analysis capabilities for scanned documents and forms. OMR Software Definition
Saturday, January 16, 2010
OCR Software and Image Processing
OCR Software and Image Processing
Why is image processing so important when utilizing Optical Character Recognition Software?
In order to get the highest possible accuracy with your OCR Application, the recognition process needs to have a clean image to examine. The most important are auto-orientation, deskew and despeckle. The Auto-orientation process examines tha page, and makes sure it is oriented correclty for the whole recognition process. Deskew examines the page for any skewing, whcih may occur during the scan process, and "rights" the page to make sure the text is inline throughout the page. Despeckle takes away any speckles on the page that can be falsely identified as font characters, but also can be attributed to any misreads of characters.
Older documents may require other functions, such as font improvement and deshading to insure the highest possible accuracy in the overall OCR process.
Why is image processing so important when utilizing Optical Character Recognition Software?
In order to get the highest possible accuracy with your OCR Application, the recognition process needs to have a clean image to examine. The most important are auto-orientation, deskew and despeckle. The Auto-orientation process examines tha page, and makes sure it is oriented correclty for the whole recognition process. Deskew examines the page for any skewing, whcih may occur during the scan process, and "rights" the page to make sure the text is inline throughout the page. Despeckle takes away any speckles on the page that can be falsely identified as font characters, but also can be attributed to any misreads of characters.
Older documents may require other functions, such as font improvement and deshading to insure the highest possible accuracy in the overall OCR process.
Subscribe to:
Posts (Atom)