The most prominent difference between still images and moving pictures stems from movements and variations. Thus to go from the realm of still image repositories to video databases, we must be able to deal with motion. Particularly, we need the ability to classify objects ap- pearing in a video sequence based on the movements of each object, as well as their other characteristics and features such as shape or color. By describing motion derived from motion analysis, we introduce a dual hierarchy consisting of spatial and tem- poral parts for video sequence representation. This gives us the flexibility to examine arbitrary frames at various levels of abstraction, and to retrieve the associated tem- poral information (say, object trajectories) in addition to the spatial representation. Our algorithm for motion detection uses the motion compensation component of the MPEG video encoding scheme. The algorithm then computes trajectories for objects of interest. The speci- fication of a language for retrieval of video based on the spatial as well as motion characteristics is presented.