CA2228679A1 - Surveillance systems - Google Patents

Abstract

An on-line surveillance system for monitoring and recording activity at a remote location is described.
Monitoring devices, such as cameras, are capable of continually surveying selected zones at the remote location, detecting activity such as motion, and providing an alarm to a monitoring station when motion is detected. Each camera's output is stored in dynamic memory such that activity occurring within a selected interval prior to initiation of an alarm is retained. Similarly, each camera's output recorded during the alarm period is stored in memory.

Description

Surveillance System Field of the Invention This invention relates to surveillance systems and more particularly to such systems employing cameras at a remote location with the camera output and any alarm conditions encoded and selectively reported to a monitoring station over a communications network.
Background Surveillance systems for fire and burglary detection are well known. In such systems motion and or heat detectors are strategically located in and around the premises and upon the detection of an event which causes an alarm a signal is forwarded over the telephone line to a monitoring or supervising station. An attendant at the monitoring station then contacts the local authorities to report the occurrence. If the occurrence represents a burglary it is likely that the intruders will have had time to leave the scene before the local authorities arrive. Evidence, which might be useful in identifying the intruders, will likely be limited to fingerprints.
Video surveillance in banks, convenience stores and the like is also known. In these surveillance systems cameras continually record activity at the selected location and after an incident the entire record needs to be reviewed to determine whether any information of value has been captured.
There is, therefore, a need to provide a surveillance system which has a motion detector and camera to monitor a particular location and to provide a real time alarm to a supervising station when unexpected motion is detected.
Preferably the system will be able to provide a video record of events immediately prior to the alarm as well as a record of the event itself. It is also desirable for the supervising station to have some configuration control of the cameras at the remote location such that certain cameras can be turned on and off and that the area scanned by the selected cameras can be varied. The real time recording of events at the supervising station provides a record that can be saved in the event that the cameras and/or records at the remote location are destroyed.
Summary of the Invention It is therefore an object of the present invention to provide a surveillance system having selective remote video monitoring, with motion detection, of a selected.location and. means to provide an alarm and record pre and post event camera images.
Therefore in accordance with a broad aspect of the present invention there is provided a surveillance system comprising: a capture station having monitoring means to detect and dynamically record events occurring at a selected location;a surveillance station remote from the capture station and connected thereto via a communications network, the surveillance station selectively monitoring the capture station and receiving notification of an event occurring at the selected location; and a database server connected to the capture station and the surveillance station to support operation thereof, the database having storage means to retain a record of selected events.
Brief Description of the Drawing The invention will now be described in grater detail with reference to the attached figure which is a high level system diagram.
Detailed Description of the Invention As shown in Figure 1 four separate (but interdependent) applications have been developed. These are remote or capture station, operator and/or management station also known as surveillance station, and video database.
The remote station or capture station is the software that runs on the computer to which one or several surveillance cameras send their output. An operator station or surveillance station is software which can connect to a remote station for the purpose of viewing output from cameras that are associated with the remote station. A
management station, which is a more sophisticated version of the operator station, can maintain open connections to one or several remote stations. In addition to viewing output from cameras associated with the remote stations, the management station can control and configure the remote stations (and the associated devices). The video database archives video that is correlated with events logged either by operators who are sitting at an operator station or by the software which is executing on the server at the remote site.
The remote station listens for connection requests from clients who wish to monitor output from a camera that is hosted at that station. When a connection request is received, the remote station determines whether the connection attempt is valid or invalid. If the connection attempt is determined to be valid, a Transmission Control Protocol (TCP) connection is opened to the connecting client and initialization data is sent to the client. This data consists of the list of those cameras hosted at the remote station that the connected client has access privileges for.
If the connecting client is an ordinary user, the client can select cameras for viewing. Once selected, output from the cameras is sent to the operator station. If the connecting client is a privileged user, additional actions are available to the client.
The server listens for incoming connections. When a connection request is detected, the server determines the validity of the request. If the request is accepted, the server opens a TCP channel to the client.
The remote station maintains a database of users. When a connection request is made, the client that issues the request must submit a username and password. If the username-password combination entered by the client is not in the database, the connection attempt is rejected. If the username-password combination is in the database, the user connection request is accepted. Information in the user record corresponding to the client that issued the connection request is used to determine the access level of the connection. For example, users may ~e restricted'to viewing output from a subset of the cameras that are hosted at the remote station.
Apart from the security represented by the login sequence, additional mechanisms for securing the remote station from hostile connections will be implemented. IP
filtering is a rudimentary form of security that can be implemented at no cost. Authentication headers will be used to create an additional security layer.
Once a client connection is accepted, a TCP connection to the client is created. The TCP connection is added to a table of connected clients. The connection is encapsulated in a class of client connections that receive and dispatch requests from a connected client.
The client connection on the server handles a11 communications with the remote client. The remote client issues a request that is received by the client connection.
The client connection parses the request and dispatches it to the appropriate handler. The client connection maintains a table of cameras that the client is monitoring and manages the flow of data from the camera to the client.
There are different classes of clients categorized by access level and other characteristics. In this embodiment, two kinds of clients are defined - ordinary users and privileged users. Ordinary users are restricted to viewing output from cameras but cannot modify the state of the cameras hosted by the remote station or the state of the remote station.
Privileged users on the other hand are permitted to configure the remote station and the devices that are associated with it. Privileged users can also monitor activity on the remote station and take corrective action if it is required. For example, if an ordinary client is monitoring output from too many cameras and thereby choking the flow of video data to other clients or to a management station, the privileged user can remove devices from the list of devices that the offending client is monitoring.
The command channel is a TCP connection that is used to send requests from the client to the remote station. The requests that can be sent over this channel will vary with the type of client.
Ordinary users can send requests to view the output from a particular camera, to stop viewing output from a particular camera and to logout out from the remote station.
2o Ordinary users cannot send commands that would alter the state of any of the monitoring devices.
The command channel will also be used for transmission of statistics that are relevant for flow control. Two kinds of data are relevant. The average time for a block of video data to travel over the network from sender to receiver and the amount of data loss (as a ratio of packets sent to packets received). This information will be used to adjust the rate at which the sender transmits data to the receiver.
A client connected to the remote station can view output 30 from one or several cameras hosted by the remote station.
The output from a camera is transmitted to the client over a video channel. For applications of the system to surveillance and security it is important to incorporate security and caching into transmission of video.

The payload is encrypted using an implementation of the public domain "Blowfish" algorithm. The Blowfish algorithm is a private key encryption scheme. To enhance the security of the system, the keys are replaced on a periodic basis. To accomplish this in a secure manner, a scheme for key management has been implemented.
For applications to security it will be necessary to cache output from each camera so that in the event that an l0 unusual condition or unexpected event is detected prior footage is available for review. The amount of pre-event video that is recorded is under the control of the end user.
Likewise, the amount of post-event video that is recorded is also under the control of the end user. For most situations, a fifteen second interval prior to the occurrence of an alarm appears to be adequate. Once an alarm has occurred recording will continue until there is a fifteen second interval that is free of alarms. The event video that is recorded at the remote site is stored at the 20 same frame rate and image quality as the video that is captured from the cameras tethered to the video server. In particular, the video that is transmitted to the operator station for viewing by central station operators can be of a lower quality and frame rate than the event video stored at the remote site. The event video that is stored at the remote site is downloaded to a video database for correlation with other event-related material.
The video channel will be implemented using the TCP
protocol. The most significant implementation related issue 30 is the reduction of video "fitter", i.e. the uneven temporal spacing of video frames due to varying network conditions.
This problem has been addressed by adjusting the video throughput from the video server to the current load on the network.
Motion detection is implemented on the remote station.
The parameters that determine the mode of operation of the mechanism for motion detection can be assigned default values or can be set remotely by a privileged user who is connected to the remote station through a management station.
This motion detection technique consists of six (6) steps that are performed sequentially. These steps, listed in order of execution, are as follows.
1. Acquisition of 3 sequential images.
2. Change filtering 3. Edge detection 4. Perimeter filter 5. Proportion filter 6. Temporal filter Data representative of three images, image I at time t ( I ( t ) [ i, j ] ) , image I at t ime t - 1 ( I ( t - 1 ) [ i, j ] ) , and image I at time t - 2 (I(t - 2)[i,j)) are digitally stored in such a way that each image can be accessed independently of each other.
Data I ( t ) [ i, j ] is compared to I ( t-1 ) [ i, j ] and I ( t-2 ) [ i, j ] . When pixels I ( t ) [ I, j ] and I ( t-1 ) [ i, j ] , and I(t)[i,j] and I(t-2)[i,j] differ by a predefined threshold f, a "1"is recorded in matrix d[i,j]. If the difference is not greater than f then a "0" is recorded at the corresponding position in matrix d[i,j]. This statement can be described by:
IF (ABS (I (t) [i, j ] - I (t-1) [i, j ] ) > f) AND (ABS (I (t) [i, j ) -I(t-2)[i,j]) > f) THEN d[i,j] - 1 ELSE d[i,j] - 0 where ABS means the absolute value.
Matrix d is put through an edge-detection algorithm.
This algorithm stores the edge data in matrix e. This process acts to separate different components in the image so that they can later be measured.
Each of the components that were separated in Edge detection is then measured for the length (in pixels) of their perimeter. If the component is not larger than pre-defined size p then it is removed from matrix e.
The height-by-width proportions of remaining components are then compared to a user-defined range r. If the components height-by-width proportions do not fall within the specified range, it is removed from matrix e. This statement can be described by:
IF (r * width) > height THEN remove component If any components remain the algorithm records this in the form: Event at time t equals true.
If event at t-2, event at t-1, and event at t are a11 true then an alarm is considered to have occurred.
Prior to transmission over the network, data from the cameras is encoded in order to reduce the amount of data that must be sent.
A smoothing filter is applied to the image prior to compression to eliminate noise introduced by the cameras.
Various techniques for compression are being employed to transmit intra- and inter-frame information.
3o Still images are compressed by first applying a wavelet transform to the image intensities and then encoding using an adaptive arithmetic encoder.
Moving images are compressed using either unconditional or conditional refreshes.

Under the unconditional refreshing scheme each frame is compressed as though it were a still image and not part of a video. This is the compression scheme that is preferred for security applications since in those applications image quality is more important than the transfer rate.
Under the conditional refreshing compression scheme, most frames are first compared to the preceding frame. The difference between the current frame and the preceding frame is calculated, compressed and transmitted. If there is to little difference between the current frame and the preceding frame, the amount of information that needs to be transmitted over the network is minimal thus significantly reducing bandwidth consumption in the case of scenes that are largely static. At the same time, the apparent transfer speed is significantly increased. Sending key frames accomplishes resynchronization with the scene at periodic intervals. Key frames are treated as still images and are compressed without reference to surrounding frames.
The last operation in the processing pipeline prior to 20 transmission is bit packing of the encoded data.
The remote station maintains a record of connections including host IP and port number, user identity, time and duration of connection, and time and duration of connection to each monitoring device to which the client has connected.
In addition, a record is kept of alarm related events. The time of onset and the duration of alarms are recorded. Other information that is associated with an alarm is the camera that triggered the alarm and the zone (an operator defined sub-field of the entire field of view of the camera) in 30 which the alarm condition was detected.
For applications to security, the principal criteria of the remote or capture station are reliable operation and image quality.
The operator station is software used by security operators and other users to connect to the remote stations that host the monitoring devices. The software used by privileged users to connect to remote stations extends the operator station. To distinguish it from the software used by ordinary users, the software used by privileged users will be called a management station. The operator station has limited functionality. From a operator station, a user can login to a remote station, select devices (cameras) to monitor, receive and view data from the monitoring devices, record data received and logout from the remote station. The l0 operator station will receive alarms that are generated at the remote sites being monitored by the operator. Once received, the alarms can be handled in different ways. In particular, a facility is provided to enable the operator to capture event video and retransmit to another location.
During login, the only actions enabled on the operator station are login and quit. The user is prompted to enter a username and password.
Once entered, the login information is sent to the remote station for validation. If the login attempt is 20 successful, a list of cameras that the operator station can connect to is sent to the operator station from the remote station. Once received, a select button is enabled on the operator station. If no response is received from the login station within a specified period of time, the login times out. If the login attempt is rejected, no other actions are enabled at the operator station except those that are enabled on start up, quit and login.
On successful login to a remote site, the user can select from a list of cameras to monitor. The list that is 30 presented to a particular user will depend on the login information that was sent during connection establishment.
Once a camera is selected, video output from that camera will be sent to the operator station until the camera is deselected. A single operator station can open connections to multiple cameras hosted on the same remote station. The upper limit will be set by the operator station and/or by the processing load on the remote station. Privileged users will be given priority over ordinary users in the event that video channels are closed because of the load on the remote station.
For different applications it may be necessary to implement other image operations. Among those that have been considered are edge enhancement and smoothing.
The maximum number of video windows that can be open l0 simultaneously and the manner of switching between views if a11 windows cannot be simultaneously displayed are implementation issues.
The management station is an operator station with enhanced capabilities. Like the operator station, the management station can connect to multiple remote stations.
Once connected, many more actions are enabled on the management station than are enabled on the operator stations. Privileged users use the management stations and the following describes functionality of the management 20 station that is not inherited from the operator station. In general, the management software can connect to multiple sites, query the status of devices hosted at remote sites, monitor the state of remote sites and configure and control remote sites and the devices associated with them.
Each management station has access rights to a set of remote stations. From the management station an authorized user can login to any of the remote stations that are linked to the management station. The login process is identical to the sequence of steps required to login to a remote station 30 from an operator station.
The management station permits authorized users to configure and control remote devices hosted by remote stations. Some of the management functions that are specific to cameras are listed below.

A privileged user will be able to start and stop devices from the management station. In contrast, ordinary users can only elect to receive output from devices that are live. Ordinary users do not have the power to activate or deactivate devices that are hosted on a remote station.
A privileged user can configure a camera to scan a rectangular sub-region of the entire field of view. This facility will be useful in cases where events of interest are known to occur in one region and that activity in. the l0 remainder of the scene is of no consequence.
The rate at which frames are captured from the device (capture rate) and the rate at which frames are delivered to the network (transmission rate) are both remotely configurable.. A possible use for this would be to capture frames at a constant rate but deliver frames at a low rate unless an unusual condition is detected. In case an unusual condition is detected the frames can be delivered as fast as they are being captured. This strategy would conserve bandwidth for transmission of data during the critical 20 periods when an unusual condition has been detected.
The management station will permit authorized users to set properties of the cameras that are being monitored such as:
brightness, contrast and white balance. The resolution of the image that is being sent can also be varied within certain limits. The upper limit is determined by the capabilities of the camera. The lower limit is determined by properties of the human visual system. The reason for allowing variation of the resolution is the same as the reason for allowing variation of the frame rate. In order to 30 conserve bandwidth, frames can be transmitted at a low resolution if an unusual condition has not been detected. As frames are continuously cached at a high resolution no information is lost if an unusual condition is detected.
When an unusual condition has been detected frames will be transmitted at a high resolution until the unusual condition disappears.
The management station can implement various types and levels of security. IP masks can be defined for each remote station that the management station is linked to. By defining a mask, connections from hosts with an IP that does not match the mask are refused. This strategy will not prevent unauthorized access by IP spoofing.
The management station can obtain information about clients that are connected to the remote station. This information will include host IP and port number, user identity, remote devices that are being monitored and time and duration of connection. In the event that suspicious activity is detected, the management station can terminate the connection to the suspect client. In addition, the management station can take control of the TCP connection to the client and thereby gain remote access to the operator station.
The management station can monitor live connections. In addition, it can view the logs that are maintained at each remote station.
The management station can remotely monitor the state of each device associated with each of the remote stations to which it is linked. The principal purpose of this facility will be to perform remote diagnostics in the event that a device malfunctions.
As part of the monitoring process, the management station will send a heartbeat to each of the remote devices that are live. If a device fails to respond, the device is offline for an unknown reason. This would constitute an exceptional condition requiring further action.
The management station can control the image-processing pipeline, i:e. the various operations that are applied to a frame before it is sent over the network. The management station can specify the filter that is to be used for smoothing an image to remove random noise.
The management station can specify the algorithm that will be used to remove noise introduced by the scanning process and to remove variations in the image caused by regular fluctuations in ambient conditions such as fluorescent lighting.
The management station can specify acceptable tradeoffs between compression ratio and speed for compression of still and moving images from each camera.
The management station allows authorized users to set the parameters that control the mode of operation of motion detection on the remote stations. The parameters that can be set are:
a) sensitivity level;
b) definition of sub-regions of the field of view; and c) motion estimation.
Each parameter can be assigned a value manually or configured automatically and dynamically.
Motion detection in the last instance demands that a decision be made as to whether a change that has been calculated between one frame and preceding frames is significant or not. Some way is needed for measuring the significance of a difference. This is the threshold value.
Differences that exceed the threshold value are deemed to be significant whereas differences that do not exceed the threshold value are not deemed to be significant.
The management station permits authorized users to define zones of interest that are rectangular sub-regions of the entire field of view of a particular camera. The threshold for motion detection can be defined independently for each zone. This permits the management station to focus attention on zones in which exceptional events are most likely to occur.

In addition to motion detection, primitive judgements concerning the direction of motion and the size of the projected area that is moving can also be made. However, computing this information places an additional load on the processor of the remote station. An authorized user may decide to enable or disable this facility on the basis of other factors.
The management station can record images from any of the cameras to which it has access. The management station l0 will archive recorded sequences and will have the capacity to playback previously recorded sequences.
The database system is designed to record significant events that occur while the application is running, as well a-s supply a central repository of information needed by the application. Additionally, the database system serves to manage video recordings of alarms to allow easy and consistent access by users of the software to these recordings.
The structure of the surveillance system consists of 20 stations which have cameras connected to them (servers).
These stations are connected to other computers called clients which receive information from the servers when motion is detected. Additionally, video can be sent from a server to these clients on a real-time basis.
The database system is based on a piece of software that runs on a computer also connected to the other stations of the system. This is called the database server. It provides centralized access to the following pieces of information:
- User logs in to camera server - Connection closed - Alarm detected - Camera server startup - Camera server shutdown - Video file deleted These events are recorded on the camera servers and clients and stored in local log files. These files are periodically downloaded from these servers and clients by the database server and analyzed to derive a coherent record of events that occurred in the system. These records are maintained on the database server.
The database system also manages a central repository to of video files and records what videos are archived and may be deleted from this storage area. Since these video files are initially stored on the camera server where the alarm occurred, the database server downloads videos from the camera servers on a periodic basis. It derives knowledge of what video files exist from the log files mentioned earlier.
Because these files are potentially very large the database server only downloads them when activity on the network is at a minimal level to ensure it does not interfere with other operations on the network.
20 The users of this software are responsible for archiving the videos stored on the database server on a regular basis using the backup medium of their choice. This process is needed because disk space is limited and the size of these files means that it can be used up quite quickly.
Although the database server does not handle the backup process, it manages the video files to allow the user to specify which files are false alarms (and may be deleted , which ones are to be kept on the database server, and which ones are to be archived. By moving each of these files to 30 different directories, accidental deletion of files can be avoided. The database server provides an interface to manage this.
The database server also maintains a record of what camera servers are at a given location, and the properties of each of those cameras are stored. Certain information is needed by a client in order to connect to a camera server.
This is also maintained by the database server which supplies this information to the clients on demand.
When an alarm occurs the client will supply information about who to contact to inform of the alarm, if necessary.
This information is maintained on the database server and can be sent on demand to the client if it requests it.
A11 the information stored by the database can be browsed and altered by users responsible for supervising the l0 operation of the surveillance system. A software application allows these users to view a11 these recorded events and the videos associated with them (assuming these video files are still on the server and have not been archived or deleted by the users).
The database will also occasionally need to be changed to reflect modifications to the system (a new camera is added to a camera server, for example). The database server also provides an interface to modify this information by authorized users.
2o While particular embodiments of the invention have been described and illustrated it will be apparent to one skilled in the art that numerous changes can be made without departing from the basic concept. It is to be understood that such changes will fall within the scope of the invention as defined by the appended claims.

Claims (15)

1. A surveillance system comprising:
a capture station having monitoring means to detect and dynamically record events occurring at a selected location;
a surveillance station remote from said capture station and connected thereto via a communications network, said surveillance station selectively monitoring said capture station and receiving notification of an event occurring at said selected location; and a database server connected to said capture station and said surveillance station to support operation thereof, said database having storage means to retain a record of selected events.

2. A surveillance system as defined in claim 1 wherein said monitoring means comprises one or more cameras.

3. A surveillance system as defined in claim 2 wherein said one or more cameras operates as a motion detector.

4. A surveillance system as defined in claim 3 wherein said motion detector examines sequential frames of camera images and provides an alarm indicating the occurrence of an event if said sequential images show a change above a threshold.

5. A surveillance system as defined in claim 4 wherein said alarm is forwarded to said surveillance system and said camera images including images obtained from a previous time interval are stored in permanent memory.

6. A surveillance system as defined in claim 5 wherein camera images are continually stored in permanent memory during the interval that an alarm condition exists.

7. A surveillance system as defined in claim 6 wherein said camera images are transported to said surveillance station upon receipt at said capture station of access information indicating that said surveillance station has proper access to said camera images.

8. A surveillance system as defined in claim 7 wherein said camera images are encoded for transfer to said surveillance station.

9. A surveillance system as defined in claim 8 wherein said encoded images are compressed.

10. A surveillance system as defined in claim 9 wherein said surveillance station has means to configure said cameras at said capture location.

11. A surveillance system as defined in claim 10 wherein said means to configure said cameras includes means to turn them on or off, and to scan the selected location.

12. An on-line surveillance system for monitoring a remote location, for detecting activity at said remote location and for generating an alarm when an activity is detected at said remote location, said surveillance system comprising:
a capture station at said remote location, said capture station having one or more cameras positioned about said remote location for selectively monitoring designated areas, and a server including a database with information respecting connections having access to said server, memory means for dynamically storing images generated by said cameras, and motion detection means to detect an unexpected occurrence at said remote location and to generate an alarm when an unexpected occurrence is detected a surveillance station connected to said capture station through a communications network, said surveillance station having access to said capture station server if permitted by said server and if permitted to selectively monitor camera output from said cameras and to receive an alarm from said server respecting the occurrence of an unexpected event; and a database server for providing an operating system to said capture station and said surveillance station and to maintain a record of camera images generated by said cameras at said remote locations.

13. A surveillance system as defined in claim 12 wherein camera images recorded at said capture station server are encoded for transfer through said network to said surveillance station.

14. A surveillance system as defined in claim 12 wherein said motion detection means examines sequential frames of camera images and if the difference in characteristics between said frames is greater than a preset threshold reports an alarm condition.

15. A surveillance system as defined in claim 14 wherein said camera images recorded in a predetermined interval prior to said alarm report are removed from said dynamic memory and stored in permanent memory and camera images recorded subsequent to said alarm report are directed to said permanent memory during the time interval that an alarm condition continues to be reported.