gps-denied-onboard/_docs/00_problem/1.2_research_prompt.md

Research this problem:

We have a lot of images taken from a wing-type UAV using a camera with at least Full HD resolution. Resolution of each photo could be up to 6200*4100 for the whole flight, but for other flights, it could be FullHD

Photos are taken and named consecutively within 100 meters of each other.

We know only the starting GPS coordinates. We need to determine the GPS of the centers of each image. And also the coordinates of the center of any object in these photos. We can use an external satellite provider for ground checks on the existing photos

The system should process data samples in the attached files (if any). They are for reference only.
 - We have the next restrictions:
    - Photos are taken by only airplane type UAVs.
    - Photos are taken by the camera pointing downwards and fixed, but it is not autostabilized.
    - The flying range is restricted by the eastern and southern parts of Ukraine (To the left of the Dnipro River)
    - The image resolution could be from FullHD to 6252*4168
    - Altitude is predefined and no more than 1km
    - There is NO data from IMU
    - Flights are done mostly in sunny weather
    - We can use satellite providers, but we're limited right now to Google Maps, which could be outdated for some regions
    - Number of photos could be up to 3000, usually in the 500-1500 range
    - During the flight, UAVs can make sharp turns, so that the next photo may be absolutely different from the previous one (no same objects), but it is rather an exception than the rule
    - Processing is done on a stationary computer or laptop with NVidia GPU at least RTX2060, better 3070. (For the UAV solution Jetson Orin Nano would be used, but that is out of scope.)

 - Output of our system should meet these acceptance criteria:
    - The system should find out the GPS of centers of 80% of the photos from the flight within an error of no more than 50 meters in comparison to the real GPS

    - The system should find out the GPS of centers of 60% of the photos from the flight within an error of no more than 20 meters in comparison to the real GPS

    - The system should correctly continue the work even in the presence of up to 350 meters of an outlier photo between 2 consecutive pictures en route. This could happen due to tilt of the plane.

    - System should correctly continue the work even during sharp turns, where the next photo doesn't overlap at all, or overlaps in less than 5%. The next photo should be in less than 150m drift and at an angle of less than 50%

    - The number of outliers during the satellite provider images ground check should be less than 10%

    - In case of being absolutely incapable of determining the system to determine next, second next, and third next images GPS, by any means (these 20% of the route), then it should ask the user for input for the next image, so that the user can specify the location

    - Less than 5 seconds for processing one image

    - Results of image processing should appear immediately to user, so that user shouldn't wait for the whole route to complete in order to analyze first results. Also, system could refine existing calculated results and send refined results again to user 

    - Image Registration Rate > 95%. The system can find enough matching features to confidently calculate the camera's 6-DoF pose (position and orientation) and "stitch" that image into the final trajectory

    - Mean Reprojection Error (MRE) < 1.0 pixels. The distance, in pixels, between the original pixel location of the object and the re-projected pixel location.


 - Find out all the state-of-the-art solutions for this problem and produce the resulting solution draft in the next format:

   - Short Product solution description. Brief component interaction diagram.

   - Architecture approach that meets restrictions and acceptance criteria. For each component, analyze the best possible approaches to solve, and form a table comprising all approaches. Each new approach would be a row, and has the next columns:

     - Tools (library, platform) to solve component tasks

     - Advantages of this approach

     - Limitations of this approach

     - Requirements for this approach

     - How does it fit for the problem component that has to be solved, and the whole solution

   - Testing strategy. Research the best approaches to cover all the acceptance criteria. Form a list of integration functional tests and non-functional tests.


Be concise in formulating. The fewer words, the better, but do not miss any important details.