How to Use Apple Vision Framework VNImageRequestHandler to Detect Text and Barcodes in Live Camera Feeds

Modern iOS applications increasingly rely on real-time visual analysis. Whether a user is scanning a QR code to authenticate a session, extracting an IBAN from a printed invoice, or parsing a driver’s license, the camera has become the primary data input mechanism. Historically, developers had to integrate bulky third-party SDKs (like OpenCV or ZXing) and manage complex memory buffers to analyze video frames. Apple has resolved this by embedding the Vision Framework directly into the OS. By coupling AVCaptureVideoDataOutput with VNImageRequestHandler, iOS engineers can execute highly performant, machine-learning-driven text and barcode detection directly on the device’s Neural Engine, ensuring absolute user privacy with near-zero latency.

The Architecture of Live Frame Processing

To process live video, you must configure an AVCaptureSession. The iPhone’s camera captures frames at 60 frames per second (FPS). You must intercept these frames before they are rendered to the screen.

You achieve this by adding an AVCaptureVideoDataOutput to your capture session and setting a delegate queue. The OS will push raw CMSampleBuffer objects containing the image data into your delegate method 60 times a second. If your image processing algorithm takes longer than 16 milliseconds to execute, the capture pipeline will choke, resulting in severe UI lag. The Vision framework solves this by offloading the heavy lifting to the specialized Neural Engine (NPU).

Constructing the Vision Requests

Before the camera begins pushing frames, you must define what the Vision framework should look for. You construct specific request objects. For this scenario, we require VNDetectBarcodesRequest and VNRecognizeTextRequest.

import Vision

class VisionAnalyzer {
    
    private var requests = [VNRequest]()
    
    func setupVision() {
        // 1. Configure Barcode Detection
        let barcodeRequest = VNDetectBarcodesRequest { (request, error) in
            guard let results = request.results as? [VNBarcodeObservation] else { return }
            for barcode in results {
                print("Found Barcode: \(barcode.payloadStringValue ?? "Unknown")")
            }
        }
        // Restrict to specific symbologies to save CPU cycles
        barcodeRequest.symbologies = [.qr, .code128]
        
        // 2. Configure Text Recognition (OCR)
        let textRequest = VNRecognizeTextRequest { (request, error) in
            guard let results = request.results as? [VNRecognizedTextObservation] else { return }
            for textObservation in results {
                // Request the top candidate
                if let topCandidate = textObservation.topCandidates(1).first {
                    print("Found Text: \(topCandidate.string) (Confidence: \(topCandidate.confidence))")
                }
            }
        }
        // Use the highly accurate ML-based recognition level
        textRequest.recognitionLevel = .accurate
        
        // 3. Store requests for the handler
        self.requests = [barcodeRequest, textRequest]
    }
}

Executing the VNImageRequestHandler

With the requests defined, you must intercept the camera feed. In your AVCaptureVideoDataOutputSampleBufferDelegate, you receive the CMSampleBuffer.

You must extract the raw pixel buffer (CVPixelBuffer) from the sample. Crucially, you then instantiate a transient VNImageRequestHandler for that specific frame, passing in the pixel buffer and the physical orientation of the device (to ensure the OCR engine reads text left-to-right). Finally, you instruct the handler to execute the array of requests you defined earlier.

import AVFoundation

extension VisionAnalyzer: AVCaptureVideoDataOutputSampleBufferDelegate {
    
    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
        
        // 1. Extract the raw image buffer
        guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
        
        // 2. Determine the physical orientation (assuming Portrait for this example)
        let orientation: CGImagePropertyOrientation = .right 
        
        // 3. Create the handler for this specific frame
        let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: orientation, options: [:])
        
        // 4. Dispatch the execution to a background queue to prevent blocking the camera thread
        DispatchQueue.global(qos: .userInitiated).async {
            do {
                // 5. Execute both barcode and text recognition simultaneously
                try imageRequestHandler.perform(self.requests)
            } catch {
                print("Failed to perform Vision request: \(error.localizedDescription)")
            }
        }
    }
}

Performance Considerations

While the Vision framework is heavily optimized, executing ML-based OCR 60 times a second is excessive and drains the battery rapidly. Advanced implementations should throttle the processing. Instead of instantiating a VNImageRequestHandler on every single frame in the captureOutput delegate, developers should drop frames (e.g., only process 5 frames per second) or utilize a lightweight tracking request (VNTrackObjectRequest) to follow a detected barcode across the screen, only invoking the heavy OCR request when the object stabilizes.